BW MaintenanceGuide R200

Maintenance Guide
Release 20.0
Document Version 13
9737 Washingtonian Boulevard, Suite 350

Gaithersburg, MD 20878
Tel +1 301.977.9440
WWW.BROADSOFT.COM
BroadWorks® Guide
Copyright Notice
Copyright© 2018 BroadSoft, Inc.
All rights reserved.
Any technical documentation that is made available by BroadSoft, Inc. is proprietary and
confidential and is considered the copyrighted work of BroadSoft, Inc.
This publication is for distribution under BroadSoft non-disclosure agreement only. No
part of this publication may be duplicated without the express written permission of
BroadSoft, Inc., 9737 Washingtonian Boulevard, Suite 350, Gaithersburg, MD 20878.
BroadSoft reserves the right to make changes without prior notice.
Trademarks
Any product names mentioned in this document may be trademarks or registered
trademarks of BroadSoft or their respective companies and are hereby acknowledged.
This document is printed in the United States of America.
BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2018 BROADSOFT, INC. PAGE 2 OF 261
Document Revision History
Release Version Reason for Change Date Author
14.0 1 Updated document for re-branding. March 15, 2006 Robb Surridge
14.0 1 Reformatted styles for re-branding. April 24, 2006 Roberta Boyle
14.0 1 Edited changes. June 9, 2006 Patricia Renaud
14.0 1 Added material to section 9.1 Automatic June 28, 2006 Robb Surridge
Backups for EV 31273, added sys.odbc.ini
file to backup list; added paragraph to
section 9.1.3 Restore Auto-Backup.
INFO corrected to UPDATE for EV 31765
for bwAuditAbnormalCall-Termination in
section 19.3.6.4 SIP/MGCP Traps.
Added recommendation for
check_dbpages.pl script in section 10.5.4
Update TimesTen Database Paging
Information for EV 33301.
Corrected examples in section 6.1.2 View
System Alarms for EVs 33243 and 33195.
Updated section 14 Install BroadWorks
License Files for EV 32926.
Corrected importdb.pl example in section
10.3 Database Import for EV 34367.
14.0 1 Updated document with Release 14.0 July 20, 2006 Mario Goupil
content.
14.0 1 Added index. July 21, 2006 Andrea Fitzwilliam
14.0 1 Edited document. August 24, 2006 Patricia Renaud
14.0 2 Added disaster recovery strategies, and August 29, 2006 Robb Surridge
minor textual updates to correct EV 34678.
14.0 2 Added CommonPersistency input channel September 11, 2006 Maxime Larose
parameters.
14.0 2 Edited changes. October 17, 2006 Patricia Renaud
14.0 3 Added new section for EV 37043, (section March 15, 2007 Roberta Boyle
21.28 Software Manager Utilities).
Changed note for EV 37037 in section
8.1.1 Display Full Details about Active
Software Versions and added text about
-showPatch option in section 21.5
bwshowver.
Added warning in section 21.23 stopbw for
EV 39877.
14.0 3 Edited changes. March 16, 2007 Patricia Renaud
14.0 4 Modified document based on Release 14 May 11, 2007 Tony Pilote
config-network changes.
14.0 4 Edited changes. May 21, 2007 Patricia Renaud
14.0 5 Added BEA WebLogic information. June 12, 2007 Martin Bernier
14.0 5 Edited changes and published document. June 20, 2007 Andrea Fitzwilliam
14.0 6 Corrected log file names for BEA for June 26, 2007 Martin Bernier
EV 49826.

©
14.0 6 Added section on removing peer from July 3, 2007 Roberta Boyle
redundant system for EV 50350.
14.0 6 Edited changes and published document. July 6, 2007 Andrea Fitzwilliam
14.0 7 Updated section 9.1.3 Restore Auto- July 12, 2007 Martin Bernier
Backup to indicate that the backup restore
must be executed as root when done in the
system root directory for EV 50841.
14.0 7 Updated section 20.9 config-network to July 27, 2007 Patricia Renaud
add a note for HTTP Alias CLI command
for EV 38723.
14.0 7 Edited changes and published document. July 27,2007 Patricia Renaud
14.0 8 Updated section 5.3.3 Enable TimesTen August 30, 2007 Roberta Boyle
Logs for EV 49372.
14.0 9 Added missing CMS-related information in September 18, 2007 Tony Pilote
section 4 BroadWorks Processes.
14.0 9 Edited changes and published document. September 18, 2007 Patricia Renaud
14.0 10 Added a procedure for importing a October 1, 2007 Tony Pilote

database from a server with a different OS.
14.0 10 Updated section 20.9 config-network for October 9, 2007 Roberta Boyle
EV 50203.
14.0 10 Added section 22.2 Adjust BroadWorks October 9, 2007 Roberta Boyle
Daylight Savings Time for EV 50504.
14.0 10 Updated section 14 Install BroadWorks October 16, 2007 Roberta Boyle
14.0 10 Changed DevDebug to Debug in section October 16, 2007 Roberta Boyle
5.1.1 Log Severity for EV 53781.
14.0 10 Edited changes and published document. October 29, 2007 Andrea Fitzwilliam
14.0 11 Updated BEA Java GC logs information. February 8, 2008 Martin Bernier
14.0 11 Edited changes and published document. February 12, 2008 Andrea Fitzwilliam
14.0 12 Added section 9.1.1 Configure Basic February 13, 2008 Roberta Boyle
Maintenance Tasks for EV 59029.
14.0 12 Updated section 5.1.7 Media Server for February 20, 2008 Roberta Boyle
EV 59375.
14.0 12 Modified section 9 Scheduled Maintenance March 6, 2008 Jean-Francois

Tasks to add CDS and EMS daily DbMaint Dignard
task; also added section 21.12
getDbSize.pl for EV 53221.
14.0 12 Created section Optimization TimesTen March 13, 2008 Roberta Boyle
Checkpoint for EV 59930.
14.0 12 Edited changes and published document. March 24, 2008 Andrea Fitzwilliam
14.0 13 Created section 10.5.7 TimesTen April 2, 2008 Roberta Boyle

Connection Optimization for EV 60233.
14.0 13 Added section 12.2.3.4 Xtended Services April 4, 2008 Martin Bernier
Platform Replacement.
14.0 13 Edited changes and published document. April 16, 2008 Andrea Fitzwilliam

©
14.0 14 Added a description of all the input May 1, 2008 Tony Pilote
channels used by BroadWorks.
14.0 14 Edited changes and published document. May 2, 2008 Andrea Fitzwilliam
14.0 15 Added Profile Server impacts. May 27, 2008 Yves Racine
15.0 1 Updated document for Release 15.0. July 9, 2008 Mario Goupil
15.0 1 Edited changes since Release 14.0, July 16, 2008 Andrea Fitzwilliam
removed references to CMS, and
published document.
15.0 2 Added configuration of CCR allowed proxy October 7, 2008 Frederic Lelievre
URLs in config-network tool and changed
Apache/Tomcat log files locations.
15.0 2 Modified startFileRep.sh section to October 9, 2008 Joël Collin

highlight its replacement utility, frepctl.
15.0 2 Added section 17 Change BCCT Listening October 31, 2008 Yves Racine
Ports for EV 62498.
15.0 2 Edited changes and published document. November 21, 2008 Andrea Fitzwilliam
15.0 3 Updated section 11.5.2 Element November 28, 2008 Goska Auerbach
Management System for EV 65145.
15.0 3 Added section 21.14.1 importdb.pl on December 2, 2008 Goska Auerbach

Element Management System for
EV 81954.
15.0 3 Updated section 12 Disaster Recovery January 14, 2009 Roberta Boyle
Strategy for EV 87423.
15.0 3 Added section 11.1.3 TimesTen January 14, 2009 Frederic Lelievre
Replication Port Number.
15.0 3 Edited changes and published document. February 5, 2009 Andrea Fitzwilliam
15.0 4 Updated section 11.5.1 Application Server, March 12, 2009 Patrick Nadeau
Network Server for EV 48848.
15.0 5 Updated server recovery procedure to April 16, 2009 Mark Kushnir
include information on file synchronizing.
15.0 6 Made minor change to section 9.1.1 May 15, 2009 Goska Auerbach
Configure Basic Maintenance Tasks for
EV 89729.
16.0 1 Removed the manual step of importing May 25, 2009 Yves Racine
data from peer as this is now automatically
done by the script config-redundancy in
section 11.5.3 Profile Server.
16.0 1 Updated document for Release 16.0. May 25, 2009 Mario Goupil
17.0 1 Updated document for Release 17.0 and October 28, 2009 Roberta Boyle
added section 20 BroadWorks Application
Server Time Zone Support for EV 101825.

©
17.0 1 Updated section 10.5.1 Change TimesTen February 9, 2010 Goska Auerbach
Database Size for EV 106025.
17.0 1 Updated document for Release 17.0. March 22, 2010 Mario Goupil
17.0 2 Removed section on Optimization April 27, 2010 Goska Auerbach

TimesTen Checkpoint for EV 110711.
17.0 2 Updated section 12.2.3 BroadWorks April 30, 2010 Goska Auerbach
Server Restoration Procedure for
EV 109374.
17.0 3 Updated section 19.1.1.2 Healthmon File July 20, 2010 Goska Auerbach
System Check for EV 112653.
17.0 4 Updated section 19.3 BroadWorks SNMP August 4, 2010 Tony Pilote
Trap Monitoring for EV 115577.
17.0 4 Added clarification for maintenance tasks August 9, 2010 Tony Pilote
in section 9 Scheduled Maintenance Tasks
for EV 115892.
Updated sections 11.1.2 TimesTen
Replication Monitoring and 11.2.2 MySQL
Replication Monitoring for EV 112000.
17.0 4 Reviewed changes. September 9, 2010 Patricia Renaud
17.0 4 Added section 9.10 Service License September 10, 2010 David Moriconi
Collect and updated document for
EV 114437.
17.0 4 Made minor editorial changes. September 14, 2010 Goska Auerbach
17.0 4 Updated section 10.5.1 Change TimesTen September 24, 2010 Goska Auerbach
17.0 4 Updated document with Release 17.sp1 September 29, 2010 Frederic Lelievre
information.
17.0 4 Edited changes and published document. November 1, 2010 Andrea Fitzwilliam
17.0 5 Updated section 10.3 Database Import for November 2, 2010 Goska Auerbach
EV 120645.
17.0 5 Updated section 22.2 Adjust BroadWorks December 10, 2010 Mario Goupil
Daylight Savings Time for EV 123592.
17.0 5 Edited and published document. January 6, 2011 Patricia Renaud
17.0 6 Updated section 10.5.6 Import TimesTen January 24, 2011 Mario Goupil
Datastores between Different Operating
Systems for EV 114783.
17.0 6 Updated section 21.12 dbUpdateStats.sh January 31, 2011 Tony Pilote
for EV 124825.
Updated section 12.2.3.3 Network Server,
Application Server, and Messaging Server
Replacement for EVs 113264 and 122851.
Updated section 11.5.1 Application Server,
Network Server, and Messaging Server for
EV 116345.

©
17.0 6 Edited changes and published document. March 8, 2011 Jessica Boyle
17.0 7 Updated section 7.10.1 Healthmon for March 16, 2011 Hang Tran
EV 129579.
17.0 7 Removed section about recreating the April 19, 2011 Mario Goupil
database by executing the nms.sh script
for EV 119909.
17.0 7 Added section 22.3 Change Media Server May 3, 2011 Mario Goupil
Kernel Selection on Linux to add steps to
change kernel selection for EV 121030.
17.0 7 Updated section 11.5.2 Element May 9, 2011 Mario Goupil

17.0 7 Updated section 19.1.6.1 Collecting Heap June 9,2011 Mark Kushnir
Size Indicator for EV 143495
17.0 7 Updated section 9.1.1 Configure Basic June 14, 2011 Goska Auerbach
17.0 7 Edited changes and published document. June 27, 2011 Jessica Boyle
17.0 8 Removed section 21.12 db.UpdateStats.sh July 18, 2011 Mario Goupil
and updated section 9.4 Database
Statistics Update for EV 124825.
17.0 8 Edited changes and published document. July 27, 2011 Jessica Boyle
17.0 9 Updated section 16 Change IP Address or August 30, 2011 David Moriconi
Host Name for EV 148587.
17.0 9 Added warning in section 10.5.1 Change September 27, 2011 Mario Goupil
TimesTen Database Size for EV 145227.
17.0 9 Corrected CLI reference from NetServ to September 29, 2011 Mario Goupil
NetworkServer for EV 150714.
17.0 9 Added CDS/CCRS restoration procedure September 29, 2011 François Isabelle
in section 12.2.3.6 Call Detail Server/Call
Center Reporting Server for EV 125022.
17.0 9 Updated section 12.2.3 BroadWorks September 30, 2011 David Moriconi
EV 144853.
17.0 9 Updated section 21.19 resizeDSN for October 12, 2011 Mario Goupil
EV 151585.
18.0 1 Updated document for Release 18.0. October 16, 2011 Mario Goupil
18.0 1 Updated section 16 Change IP Address, October 19, 2011 Goska Auerbach
Host Name, FQDN Name, Gateway, or
DNS Server for EV 144590.
18.0 1 Edited changes and published document. October 21, 2011 Andrea Fitzwilliam
18.0 2 Updated section 19.1.6.1 Collecting Heap October 24, 2011 Mario Goupil
Size Indicator for EV 147625.
18.0 2 Edited changes and published document. November 4, 2011 Jessica Boyle
18.0 3 Updated section 10.5.6 Import TimesTen November 8, 2011 Frederic Lelievre
Systems to include UID and PWD
parameters.

©
18.0 3 Added section 12.2.3.7 Element November 22, 2011 David Moriconi
Management System to cover the disaster
recovery process for the EMS for
EV 130651.
18.0 3 Updated section 10.5.6 Import TimesTen November 30, 2011 Frederic Lelievre
Systems to define UID and PWD
parameters as lowercase.
18.0 3 Updated section 12.2.3.7 Element January 11, 2012 David Moriconi
18.0 3 Updated section 16 Change IP Address, January 16, 2012 David Moriconi
18.0 3 Edited changes and published document. January 25, 2012 Jessica Boyle
18.0 4 Updated section 12.2.3 BroadWorks February 1, 2012 Goska Auerbach

EV 130651.
18.0 4 Edited changes and published document. February 16, 2012 Jessica Boyle
18.0 5 Updated section 11.5 Add New Peer to March 21,2012 François Isabelle
Redundant System for EV 162349.
18.0 5 Removed references to the Conferencing March 27, 2012 François Isabelle
Server for EV 154846.
18.0 5 Edited changes and published document. April 10, 2012 Jessica Boyle
18.0 6 Updated section 12.2.3.3 Network Server, April 17, 2012 Mario Goupil
Replacement to add rsync command for
IpDeviceConfig files for EV 164187.
18.0 6 Edited changes and published document. April 27, 2012 Jessica Boyle
18.0 7 Updated section 11.5 Add New Peer to June 15, 2012 François Isabelle
Redundant System for EV 162349.
18.0 7 Updated section 21.8 config-hostname for July 18, 2012 Mario Goupil
EV 169451.
18.0 7 Updated section 12.2.3.3 Network Server, August 2, 2012 Goska Auerbach
Replacement for EV 165887.
18.0 7 Updated section 16 Change IP Address, August 6, 2012 François Isabelle

18.0 7 Edited changes and published document. August 24, 2012 Jessica Boyle
18.0 8 Updated section 9.1.1 Configure Basic September 5,2012 François Isabelle
18.0 8 Reviewed changes. September 5, 2012 Patricia Renaud
18.0 8 Updated section 12.2.3.2 Network Server, September 13, 2012 François Isabelle
18.0 8 Reviewed changes. September 14, 2012 Goska Auerbach

©
19.0 1 Added section 10.6.3 Regain Disk Space September 18, 2012 Hang Tran
Used by MySQL InnoDB Tablespace Files.
19.0 1 Updated section 9 Scheduled Maintenance September 28, 2012 Goska Auerbach
Tasks for EV 175534.
19.0 1 Update section 12 Disaster Recovery October 1, 2012 François Isabelle

19.0 1 Updated section 7.9 Restore BroadWorks October 15, 2012 Goska Auerbach
Server to In-service for EV 176540.
19.0 1 Added Service Control Function (SCF) October 21 2012 Mario Goupil
server.
19.0 1 Updated section 12 Disaster Recovery October 24, 2012 David Moriconi
19.0 1 Updated section 9.1.3 Restore Auto- October 25, 2012 David Moriconi
Backup for EV 148461.
19.0 1 Modified section 10.6.3 Regain Disk Space November 2, 2012 David Moriconi
Used by MySQL InnoDB Tablespace Files
for EV 164733.
19.0 1 Edited changes and published document. November 16, 2012 Patricia Renaud
19.0 2 Added bwLoadHarderning.pl tool definition. November 21, 2012 Frederic Lelievre
19.0 2 Adding new peerctl commands in section November 30, 2012 Frederic Lelievre
21.16 peerctl for EV 113764.
19.0 2 Edited changes and published document. March 1, 2013 Joan Renaud
19.0 3 Updated section 9.2 Automatic Disk June 17, 2013 Jerry Zhang
Cleanup for EV 189686.
19.0 3 Updated section 11.5.1 Application Server, August 25, 2013 Mario Goupil
Network Server for EV 197735.
19.0 3 Updated section 9 Scheduled Maintenance September 18, 2013 Jean-Francois

Tasks to add Service Control Function Dignard
(SCF) Server for EV 201523.
19.0 3 Edited changes and published document. September 20, 2013 Jessica Boyle
19.0 4 Added section 21.29 configdctl for September 24, 2013 Frederic Lelievre
EV 202216.
19.0 4 Edited changes and published document. October 23, 2013 Jessica Boyle
20.0 1 Updated section 8.3 Switch Active October 29, 2013 Jerry Zhang
Software Version for EV 198263. Updated
section 12.2.3.1 Media Server
20.0 1 Updated sections 19.2.1.3.1 Real-Time December 4, 2013 Goska Auerbach

Memory Usage and 19.2.1.4.2 Transmit
RTP for EV 206499.
20.0 1 Edited changes and published document. December 13, 2013 Joan Renaud
20.0 2 Updated document for the WebRTC December 14, 2013 Mario Goupil
Server, Messaging Server, and Sharing
Server.
Removed references to the Call Detail
Server.

©
20.0 2 Edited changes and published document. January 27, 2014 Patricia Renaud
20.0 3 Updated section 21.23 stopbw for January 29, 2014 Frederic Lelievre
EV 191707.
20.0 3 Edited changes and published document. February 4, 2014 Jessica Boyle
20.0 4 Updated section 21.16 peerctl for March 7, 2014 Mario Goupil
EV 218395.
20.0 4 Updated all references to the Sharing March 26, 2014 Joan Renaud
Server (USS) and the Messaging Server
(UMS).
Edited changes and published document.
20.0 5 Updated section 16.4 Update Application May 3, 2014 Mario Goupil
Server Call Processing Routing Information
for EV 221793.
20.0 5 Edited changes. May 30, 2014 Jessica Boyle
20.0 5 Made minor editorial changes and July 25, 2014 Joan Renaud
published document.
20.0 6 Added section 12.2.3.8 Execution Server November 18, 2014 Mario Goupil
20.0 6 Updated sections 22.1 Add Memory to December 2, 2014 Goska Auerbach
BroadWorks Server and 12.2.3
BroadWorks Server Restoration Procedure
for EV 240907.
20.0 6 Updated section 14 Install BroadWorks December 8, 2014 Hang Tran

20.0 6 Updated sections 12.2.3.4 Xtended December 10, 2014 Goska Auerbach
Services Platform Replacement and
12.2.3.4 Profile Server Replacement for
EV 232459.
20.0 6 Updated sections 5.1.7 Sample Debug December 30, 2014 Mario Goupil
Configuration File and 5.1.8 Platform
Daemons Logging for EV 216892.
20.0 6 Edited changes and published document. February 2, 2015 Joan Renaud
20.0 7 Updated section 22.1 Add Memory to February 5, 2015 Goska Auerbach
BroadWorks Server for EV 216897.
Updated section 9 Scheduled Maintenance
Tasks for EV 239421.
20.0 7 Updated section 10.5.1 Change TimesTen February 10, 2015 Goska Auerbach
20.0 8 Updated section 12.2.3 BroadWorks March 18, 2015 Goska Auerbach
PR-44949.

©
20.0 9 Updated the following sections for June 5, 2015 Goska Auerbach
PR-47532: 10.1.1 Manual Database
Backup, 10.3 Database Import, 10.6.2
Start and Stop MySQL Database Daemon,
and 16.13 Update EMS Redundancy
Configuration.
20.0 9 Updated section 5 Logs for PR-47819. July 9, 2015 Goska Auerbach
20.0 9 Updated section 11.1.1 TimesTen July 29, 2015 Goska Auerbach
Database Synchronization for PR-47884.
20.0 9 Updated section 10.5.6 Import TimesTen August 26, 2015 Goska Auerbach
Systems for PR-48290.
20.0 9 Edited changes and published document. October 10, 2015 Joan Renaud
20.0 10 Updated section 7.9.2 Redundant Systems October 15, 2015 Goska Auerbach
for PR-48695.
20.0 10 Updated section 10.3.1 Import Database November 11, 2015 Goska Auerbach
on Application Server, Element
Management System, Network Server, or
Messaging Server for PR-48912,
20.0 11 Updated section 11.5.1 Application Server, October 13, 2016 Goska Auerbach
PR-51605.
20.0 11 Updated section 11.5.1 Application Server, March 7, 2017 Goska Auerbach
PR-54578.
20.0 11 Updated section 12.2.3.2 Application March 16, 2017 Goska Auerbach
Server Replacement for PR-54802.
20.0 12 Updated section 19.4.2 Redundancy April 4, 2017 Goska Auerbach

Monitoring to cover the case where a
server has stopped its database replicated
toward a peer for PR-55026.
20.0 12 Updated sections 7.8.2 Redundant June 20, 2017 Goska Auerbach
Systems and 7.9.2 Redundant Systems for
PR-55512.
20.0 12 Updated section 16 Change IP Address, June 22, 2017 Goska Auerbach
DNS Server and added section 16.2
Restart BroadWorks Software Manager for
PR-55554.
20.0 12 Updated section 16 Change IP Address, July 7, 2017 Goska Auerbach

DNS Server for PR-55622.
20.0 12 Updated section 10.5.5 Compact the July 31, 2017 Goska Auerbach
Database for PR-55962.
20.0 12 Edited changes and published document. August 7, 2017 Jessica Boyle
20.0 13 Added section 16 Uninstall Obsolete Third- November 15, 2017 Goska Auerbach
Party Components for PR-55193.

©

©
Table of Contents
1 Overview....................................................................................................................................... 20
2 Summary of Changes ................................................................................................................ 21
2.1 Changes for Release 20.0 ........................................................................................................ 21
2.8 Changes from Release 13.0 ..................................................................................................... 29
3 Maintenance Recommendations ............................................................................................. 31
4 BroadWorks Processes............................................................................................................. 32
5 Logs .............................................................................................................................................. 35
5.1 BroadWorks Debug Logs .......................................................................................................... 38
5.1.1 Log Severity ...................................................................................................................... 39
5.1.2 Log Input Channels .......................................................................................................... 39
5.1.3 Input Channel Attributes Channels .................................................................................. 51
5.1.4 Enable or Disable BroadWorks Debug Logging ............................................................. 52
5.1.5 Output Channel attributes ................................................................................................ 52
5.1.6 Log Rotation...................................................................................................................... 53
5.1.7 Sample Debug Configuration File.................................................................................... 53
5.1.8 Platform Daemons Logging ............................................................................................. 54
5.2 BroadWorks UNIX Command Execution Logging .................................................................. 54
5.3 Miscellaneous Logs ................................................................................................................... 55
5.3.1 Solaris System Logs ......................................................................................................... 55
5.3.2 Linux System Logs ........................................................................................................... 55
5.3.3 Enable TimesTen Logs .................................................................................................... 55
5.3.4 Apache Logs ..................................................................................................................... 55
5.3.5 Tomcat Logs ..................................................................................................................... 55
5.3.6 Oracle Logs ....................................................................................................................... 55
6 Fault and Performance Measurements .................................................................................. 56
6.1 Alarms ........................................................................................................................................ 56
6.1.1 Open System Alarms ....................................................................................................... 56
6.1.2 View System Alarms ........................................................................................................ 57
6.1.3 View and Set CLI Alarm Backlog Size ............................................................................ 58
6.1.4 Enable and Disable Real-Time Alarms Echoing............................................................. 59

©
6.1.5 Clear Alarms ..................................................................................................................... 60
6.1.6 Close System Alarms ....................................................................................................... 60
6.2 Threshold Alarms ...................................................................................................................... 60
6.2.1 Threshold Attributes ......................................................................................................... 60
6.2.2 Configure Thresholds using CLI ...................................................................................... 61
6.2.3 Configure Thresholds through SNMP ............................................................................. 63
6.2.4 Application Server Thresholds Limitation ........................................................................ 64
6.3 Performance Measurements .................................................................................................... 65
6.3.1 Navigate through MIB Nodes........................................................................................... 65
6.3.2 View Next MIB Node ........................................................................................................ 65
6.3.3 View Current Position in MIB ........................................................................................... 66
6.3.4 View MIB Node Values .................................................................................................... 66
7 Server Management ................................................................................................................... 68
7.1 Server State Management ........................................................................................................ 68
7.1.1 Server State ...................................................................................................................... 68
7.1.2 Application State ............................................................................................................... 68
7.1.3 Get Server and Application States................................................................................... 71
7.2 Application Management .......................................................................................................... 71
7.2.1 Installation ......................................................................................................................... 72
7.2.2 Activation ........................................................................................................................... 73
7.2.3 Deployment ....................................................................................................................... 74
7.2.4 Undeployment................................................................................................................... 74
7.2.5 Deactivation ...................................................................................................................... 76
7.2.6 Uninstallation..................................................................................................................... 76
7.3 Start Server ................................................................................................................................ 76
7.3.1 Start BroadWorks ............................................................................................................. 77
7.3.2 Start BroadWorks Application .......................................................................................... 77
7.4 Stop Server ................................................................................................................................ 77
7.4.1 Stop BroadWorks ............................................................................................................. 77
7.4.2 Stop Server without Locking ............................................................................................ 78
7.4.3 Stop BroadWorks Application .......................................................................................... 78
7.5 Check for Running Processes .................................................................................................. 79
7.6 Lock Network Server, Application Server, or Execution Server.............................................. 80
7.6.1 Force Lock Server Immediately ....................................................................................... 80
7.6.2 Unlock Server ................................................................................................................... 80
7.7 Restart Server ............................................................................................................................ 81
7.8 Take BroadWorks Server Out of Service for Maintenance ..................................................... 81
7.8.1 Non-redundant Systems .................................................................................................. 81
7.8.2 Redundant Systems ......................................................................................................... 81
7.9 Restore BroadWorks Server to In-service ............................................................................... 82
7.9.1 Non-redundant Systems .................................................................................................. 82
7.9.2 Redundant Systems ......................................................................................................... 83

©
7.10 Server State Monitoring Tools .................................................................................................. 84
7.10.1 Healthmon......................................................................................................................... 84
7.10.2 Tech-support ..................................................................................................................... 86
7.11 Server Maintenance State ........................................................................................................ 87
7.11.1 View Current Maintenance State ..................................................................................... 87
7.11.2 Force Server into Maintenance State .............................................................................. 87
7.11.3 Recover Server in Maintenance State............................................................................. 87
8 Software Version Management ................................................................................................ 88
8.1 Display Active Software Version ............................................................................................... 88
8.1.1 Display Full Details about Active Software Versions ...................................................... 88
8.2 List Installed Software Versions ................................................................................................ 88
8.2.1 List Installed and Active Software Versions .................................................................... 88
8.3 Switch Active Software Version ................................................................................................ 89
9 Scheduled Maintenance Tasks ................................................................................................ 91
9.1 Automatic Backups.................................................................................................................... 93
9.1.1 Mode of Operation ............................................................................................................ 93
9.1.2 Auto-Backup File Naming Convention ............................................................................ 94
9.1.3 Restore Auto-Backup ....................................................................................................... 94
9.2 Automatic Disk Cleanup ............................................................................................................ 96
9.3 Healthmon Keep Alives ............................................................................................................. 97
9.4 Database Statistics Update....................................................................................................... 97
9.5 Registration Audit ...................................................................................................................... 97
9.6 Database Synchronization Check ............................................................................................ 98
9.7 CPU Monitoring Tool ................................................................................................................. 98
9.8 Check DB Pages ....................................................................................................................... 98
9.9 Tech-support .............................................................................................................................. 98
9.10 Service License Collect ............................................................................................................. 98
9.11 Delete Scheduled Tasks ........................................................................................................... 98
10 Database Maintenance .............................................................................................................. 99
10.1 Database Backup ...................................................................................................................... 99
10.1.1 Manual Database Backup................................................................................................ 99
10.1.2 Backup Database for Application Server, Element Management System, Call Center
Reporting Server, Network Server, or Messaging Server ............................................ 100
10.2 Database Restore.................................................................................................................... 100
10.2.1 Restore Database........................................................................................................... 100
10.2.2 Restore Database for Application Server, Element Management System, Call Center
Reporting Server, Network Server, or Messaging Server ............................................ 101
10.2.3 Extract Database Backup from Automatic Backup File................................................ 101
10.3 Database Import ...................................................................................................................... 102
10.3.1 Import Database on Application Server, Element Management System, Network
Server, or Messaging Server ......................................................................................... 103
10.4 Database Installation ............................................................................................................... 104

©
10.4.1 Recreate Application Server, Network Server, or Messaging Server Database ........ 104
10.5 Additional TimesTen Maintenance Tasks .............................................................................. 106
10.5.1 Change TimesTen Database Size ................................................................................ 106
10.5.2 Update TimesTen Database Statistics .......................................................................... 107
10.5.3 Start and Stop TimesTen Database Daemon............................................................... 107
10.5.4 Update TimesTen Database Paging Information ......................................................... 107
10.5.5 Compact the Database .................................................................................................. 108
10.5.6 Import TimesTen Datastores between Different Operating Systems .......................... 108
10.5.7 TimesTen Connection Optimization .............................................................................. 110
10.6 Additional MySQL Maintenance Tasks .................................................................................. 111
10.6.1 Update MySQL Database Statistics .............................................................................. 111
10.6.2 Start and Stop MySQL Database Daemon................................................................... 111
10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files ................................. 112
11 Redundancy Management ...................................................................................................... 113
11.1 Application Server, Network Server, and Messaging Server Database Replication ........... 113
11.1.1 TimesTen Database Synchronization ........................................................................... 113
11.1.2 TimesTen Replication Monitoring .................................................................................. 115
11.1.3 TimesTen Replication Port Number .............................................................................. 117
11.2 Element Management System Database Replication........................................................... 119
11.2.1 MySQL Database Synchronization ............................................................................... 119
11.2.2 MySQL Replication Monitoring ...................................................................................... 120
11.3 Install Cluster of Servers ......................................................................................................... 121
11.3.1 Element Management System Additional Parameters ................................................ 122
11.4 Network Server and Application Server Synchronization...................................................... 124
11.5 Add New Peer to Redundant System .................................................................................... 124
11.5.1 Application Server, Network Server, and Messaging Server ....................................... 124
11.5.2 Element Management System ...................................................................................... 129
11.5.3 Profile Server .................................................................................................................. 131
11.6 Remove Peer from Redundant System ................................................................................. 133
11.7 Recover Faulty Redundant Installation .................................................................................. 134
11.8 Post-installation Control........................................................................................................... 134
11.9 SSH/RSYNC Configuration .................................................................................................... 135
11.9.1 Create SSH Keys (Completed Automatically During Installation) ............................... 135
11.9.2 Configure SSH Between BroadWorks Servers ............................................................ 135
11.9.3 Use SSH ......................................................................................................................... 137
12 Disaster Recovery Strategy .................................................................................................... 138
12.1 Recover Server OS ................................................................................................................. 138
12.1.1 Hot Standby Servers ...................................................................................................... 139
12.1.2 On-Demand Rebuild....................................................................................................... 139
12.1.3 Incremental Backups ...................................................................................................... 140
12.2 BroadWorks Server Restoration ............................................................................................. 141
12.2.1 BroadWorks System Backup ......................................................................................... 141

©
12.2.2 Licensing ......................................................................................................................... 142
12.2.3 BroadWorks Server Restoration Procedure ................................................................. 142
13 Optional Remember Password Cookie for Administrators .............................................. 154
13.1 Modify Web.xml File ................................................................................................................ 154
13.1.1 Disable Cookie Support ................................................................................................. 154
13.1.2 Enable Cookie Support .................................................................................................. 154
14 Install BroadWorks License Files .......................................................................................... 155
15 Uninstall Obsolete BroadWorks Versions ........................................................................... 156
16 Uninstall Obsolete Third-Party Components ...................................................................... 157
17 Change IP Address, Host Name, FQDN Name, Gateway, or DNS Server ...................... 158
17.1 Configure IP Networking Information at UNIX Level ............................................................. 162
17.2 Restart BroadWorks Software Manager ................................................................................ 163
17.3 Update BroadWorks Network Configuration .......................................................................... 164
17.3.1 Restart the configuration agent ...................................................................................... 164
17.3.2 HTTP Server Network Configuration ............................................................................. 164
17.3.3 SNMP Trap Source Address Configuration .................................................................. 164
17.4 Configure Redundancy Routing ............................................................................................. 165
17.5 Update Application Server Call Processing Routing Information ......................................... 165
17.6 Update Application Server IP Address on Network Server ................................................... 166
17.7 Update Network Server IP Address on Application Server ................................................... 167
17.8 Update Network Server Routing Alias List ............................................................................. 168
17.9 Update Media Server IP Address on Network Server ........................................................... 169
17.10 Update Media Server IP Address on Application Server ...................................................... 170
17.11 Update Application Server and Network Server Address on OCS....................................... 170
17.12 Update Xtended Services Platform IP Address on Network Server..................................... 171
17.13 Update Network Server or Application Server Address on Xtended Services Platform ..... 172
17.14 Update EMS Redundancy Configuration ............................................................................... 172
17.15 Import EMS Database ............................................................................................................. 173
17.16 Update Notification Subsystem Configuration ....................................................................... 173
17.17 Update OCI Provisioning Network Access List on Profile Server ......................................... 173
17.18 Update Execution Server IP Address on Network Server..................................................... 173
17.19 Update Profile Server IP Address on Network Server .......................................................... 173
18 Change BCCT Listening Ports ............................................................................................... 174
19 Solaris DiskSuite Maintenance .............................................................................................. 175
19.1 Background .............................................................................................................................. 175
19.1.1 Submirror Status ............................................................................................................. 175
19.1.2 Replacement Disk Requirements .................................................................................. 175
19.1.3 Metadatabase Replica Status ........................................................................................ 176
19.2 Maintenance and Recovery Procedures ................................................................................ 176
19.2.1 System Running ............................................................................................................. 176
19.2.2 System Fails to Boot Properly........................................................................................ 179

©
20 BroadWorks System Monitoring ........................................................................................... 185
20.1 Platform Resource Monitoring – Key Resource Indicators ................................................... 185
20.1.1 File System Monitoring ................................................................................................... 185
20.1.2 CPU Usage Monitoring .................................................................................................. 186
20.1.3 Memory Usage/Paging Monitoring ................................................................................ 188
20.1.4 Disk Activity ..................................................................................................................... 189
20.1.5 Thread Activity ................................................................................................................ 190
20.1.6 Application Server Execution Server Heap Monitoring ................................................ 191
20.1.7 Database Size Monitoring .............................................................................................. 192
20.2 BroadWorks Performance Measurements Monitoring .......................................................... 193
20.2.1 Media Server................................................................................................................... 193
20.2.2 Network Server ............................................................................................................... 200
20.2.3 Application Server, Execution Server, and Profile Server ............................................ 205
20.3 BroadWorks SNMP Trap Monitoring ...................................................................................... 223
20.3.1 Common BroadWorks Server Traps ............................................................................. 223
20.3.2 Server Process Problems .............................................................................................. 224
20.3.3 Software Error Traps ...................................................................................................... 224
20.3.4 Media Server Traps ........................................................................................................ 225
20.3.5 Network Server Traps .................................................................................................... 225
20.3.6 Application Server, Execution Server, and Profile Server Traps ................................. 226
20.3.7 Application Server Execution Server Process Problems ............................................. 229
20.3.8 Overload Control Monitoring .......................................................................................... 229
20.3.9 Device Operational State Monitoring ............................................................................. 229
20.4 Miscellaneous Monitoring........................................................................................................ 230
20.4.1 BroadWorks System Sanity ........................................................................................... 230
20.4.2 Redundancy Monitoring ................................................................................................. 232
20.4.3 Application Server Migrated User Monitoring ............................................................... 233
20.4.4 Database Synchronization Monitoring........................................................................... 234
21 BroadWorks Application Server Time Zone Support ........................................................ 235
22 BroadWorks Scripts and Utilities .......................................................................................... 236
22.1 bwBackup.pl............................................................................................................................. 236
22.2 bwCPUMon.............................................................................................................................. 236
22.3 bwListPortInUse....................................................................................................................... 236
22.4 bwRestore.pl ............................................................................................................................ 237
22.5 bwshowver ............................................................................................................................... 237
22.6 bwTestDaemon.pl ................................................................................................................... 237
22.7 check_dbpages.pl.................................................................................................................... 238
22.8 config-hostname ...................................................................................................................... 239
22.9 config-redundancy ................................................................................................................... 239
22.10 config-ssh ................................................................................................................................. 239
22.11 dbsetup.sh................................................................................................................................ 240
22.12 getDbSize.pl............................................................................................................................. 240

©
22.13 healthmon ................................................................................................................................ 240
22.14 importdb.pl ............................................................................................................................... 241
22.14.1 importdb.pl on Element Management System......................................................... 241
22.15 installdialplan.sh....................................................................................................................... 242
22.16 peerctl ....................................................................................................................................... 243
22.17 peercmd ................................................................................................................................... 244
22.18 repctl ......................................................................................................................................... 244
22.19 resizeDSN ................................................................................................................................ 245
22.20 showrun.................................................................................................................................... 245
22.21 startbw ...................................................................................................................................... 246
22.22 frepctl ........................................................................................................................................ 246
22.23 stopbw ...................................................................................................................................... 247
22.24 synchcheck_basic.pl ............................................................................................................... 248
22.25 synchcheck_full.pl.................................................................................................................... 248
22.26 ttStatus ..................................................................................................................................... 248
22.27 bwLoadHardening.pl ............................................................................................................... 249
22.28 Software Manager Utilities ...................................................................................................... 250
22.28.1 backup-swmanager.pl ............................................................................................... 250
22.28.2 restore-swmanager.pl................................................................................................ 250
22.29 configdctl .................................................................................................................................. 250
23 Miscellaneous Maintenance Tasks ....................................................................................... 251
23.1 Add Memory to BroadWorks Server ...................................................................................... 251
23.2 Adjust BroadWorks Daylight Savings Time ........................................................................... 251
23.2.1 Description ...................................................................................................................... 251
23.2.2 Service Impact ................................................................................................................ 252
23.2.3 Solution ........................................................................................................................... 252
23.2.4 Instructions ...................................................................................................................... 253
23.3 Change Media Server Kernel Selection on Linux .................................................................. 253
Index ................................................................................................................................................... 255

©
1 Overview
Routine maintenance of the system includes procedures for all BroadWorks servers. For
questions about these procedures, contact support@broadsoft.com.
For maintenance and care of Sun equipment, see the Sun reference manuals. For
partner devices, see the manufacturer’s documentation.

©
2 Summary of Changes
This section describes the changes to this document for each release and document
version.
2.1 Changes for Release 20.0

This section describes the changes to this document for each document version.
Release 20.0, Version 13

This version of the document includes the following changes:
 Added section 16 Uninstall Obsolete Third-Party Components for PR-55193.

 Updated section 19.4.2 Redundancy Monitoring to cover the case where a server has
stopped its database replicated toward a peer for PR-55026.
 Updated sections 7.8.2 Redundant Systems and 7.9.2 Redundant Systems for PR-
55512.
 Updated section 16 Change IP Address, Host Name, FQDN Name, Gateway, or DNS
Server and added section 16.2 Restart BroadWorks Software Manager for PR-55554.
Server for PR-55622.
 Updated section 10.5.5 Compact the Database for PR-55962.

 Updated section 11.5.1 Application Server, Network Server, and Messaging Server
for PR-51605.
for PR-54578.
 Updated section 12.2.3.2 Application Server Replacement for PR-54802.

 Updated section 7.9.2 Redundant Systems for PR-48695.
 Updated section 10.3.1 Import Database on Application Server, Element
Management System, Network Server, or Messaging Server for PR-48912.

 Updated following sections for PR-47532: 10.1.1 Manual Database Backup, 10.3
Database Import, 10.6.2 Start and Stop MySQL Database Daemon, and 16.13
Update EMS Redundancy Configuration.
 Updated section 5 Logs for PR-47819.
 Updated section 11.1.1 TimesTen Database Synchronization for PR-47884.

©
 Updated section 10.5.6 Import TimesTen Datastores between Different Operating
Systems for PR-48290.

This version of the document includes the following change:
 Updated section 12.2.3 BroadWorks Server Restoration Procedure for PR-44949.

 Updated section 22.1 Add Memory to BroadWorks Server for EV 216897.
 Updated section 9 Scheduled Maintenance Tasks for EV 239421.
 Updated section 10.5.1 Change TimesTen Database Size for EV 244136.

 Added section 12.2.3.8 Execution Server Replacement for EV 241916.
 Updated sections 22.1 Add Memory to BroadWorks Server and 12.2.3 BroadWorks
Server Restoration Procedure for EV 240907.
 Updated section 14 Install BroadWorks License Files for EV 240165.
 Updated sections 12.2.3.4 Xtended Services Platform Replacement and 12.2.3.4
Profile Server Replacement for EV 232459.

 Updated section 16.4 Update Application Server Call Processing Routing Information
for EV 221793.

 Updated section 21.16 peerctl for EV 218395.
 Updated all references to Sharing Server (USS) and Messaging Server (UMS).

 Updated section 21.23 stopbw for EV 191707.

 Updated the document for the WebRTC Server (WRS), Messaging Server (UMS),
and Sharing Server (USS).
 Removed references to the Call Detail Server (CDS).

 Updated section 8.3 Switch Active Software Version for EV 198263.

©
 Updated section 12.2.3.1 Media Server Replacement for EV 205360.
 Updated sections 19.2.1.3.1 Real-Time Memory Usage and 19.2.1.4.2 Transmit RTP
for EV 206499.


 Added section 21.29 configdctl for EV 202216.

 Updated section 9.2 Automatic Disk Cleanup for EV 189686.
for EV 197735.

 Added bwLoadHarderning.pl tool definition.
 Updated section 21.16 peerctl: added new peerctl commands for EV 113764.

 Added section 10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files
for EV 164733.
 Update section 12 Disaster Recovery Strategy for EVs 175201 and 168478.
 Updated section 7.9 Restore BroadWorks Server to In-service for EV 176540.
 Added Service Control Function (SCF) Server.
 Updated section 9.1.3 Restore Auto-Backup for EV 148461.
 Modified section 10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace
Files for EV 164733.


 Updated section 9.1.1 Configure Basic Maintenance Tasks for EV 164026. The
changes were subsequently moved to section 9.2 Automatic Disk Cleanup.
 Updated section 12.2.3.3 Network Server, Application Server, and Messaging Server

©
 Updated section 11.5 Add New Peer to Redundant System for EV 162349.
 Updated section 21.8 config-hostname for EV 169451.

Replacement to add rsync command for IpDeviceConfig files for EV 164187.

 Updated section 11.5 Add New Peer to Redundant System for EV 162349.
 Removed references to the Conferencing Server for EV 154846.

 Updated section 12.2.3 BroadWorks Server Restoration Procedure for EV 130651.

Systems to include UID and PWD parameters.
 Added section 12.2.3.7 Element Management System Replacement to cover the
disaster recovery process for the EMS for EV 130651.

 Updated section 19.1.6.1 Collecting Heap Size Indicator for EV 147625.

 Updated document for Release 18.0.


©
 Updated section 21.19 resizeDSN for EV 151585.
 Updated section 16 Change IP Address or Host Name for EV 148587.
 Corrected CLI references from NetServ to NetworkServer for EV 150714.
 Added CDS/CCRS restoration procedure in section 12.2.3.6 Call Detail Server/Call
Center Reporting Server for EV 125022.

 Removed section 21.12 db.UpdateStats.sh and updated section 9.4 Database
Statistics Update for EV 124825.

 Updated section 7.10.1 Healthmon for EV 129579.
 Removed section about recreating the database by executing the _nms.sh script for
EV 119909.
 Added section 22.3 Change Media Server Kernel Selection on Linux for EV 121030.
 Updated section 11.5.2 Element Management System for EV 141355.
 Updated section 19.1.6.1 Collecting Heap Size Indicator for EV 143495.
 Updated section 9.1.1 Configure Basic Maintenance Tasks for EV 143768.

Systems for EV 114783.
 Updated section 21.12 dbUpdateStats.sh to indicate that the task may cause
provisioning to fail while running for EV 124825.
Replacement for EVs 113264 and 122851.
 Fixed the procedure related to resynchronizing a Media Server in section 12.2.3.2
Network Server, Application Server, and Messaging Server Replacement for
EV 122851.
Replacement with information about the localization backup information included in
the auto-backup file for EV 113264.
 Updated section 11.5.1 Application Server, Network Server for EV 116345.


©
 Updated section 10.3 Database Import for EV 120645.
 Updated section 22.2 Adjust BroadWorks Daylight Savings Time for
EV 123592.

 Updated document with Release 17.sp1 information.
 Updated section 19.3 BroadWorks SNMP Trap Monitoring for EV 115577.
 Updated sections 11.1.2 TimesTen Replication Monitoring and 11.2.2 MySQL
Replication Monitoring for EV 112000.
 Added section 9.10 Service License Collect and updated section 9 Scheduled

 Updated section 19.1.1.2 Healthmon File System Check for EV 112653.

 Removed section on Optimization TimesTen Checkpoint for EV 110711.

 Updated document for Release 17.0.
 Updated document for Release 17.0 and added section 20 BroadWorks Application
Server Time Zone Support for EV 101825.

This is a maintenance version of this document. This version includes the following
changes:
 In section 11.5.3 Profile Server, the manual step of importing data from peer was
removed as this is now automatically done by the script config-redundancy.
 Made minor updates.

This section describes the changes to this document for each release and document
version.

©
 Made minor change to section 9.1.1 Configure Basic Maintenance Tasks for
EV 89729.

 Updated server recovery procedure to include information on file synchronizing.

This is a maintenance version of this document. This version of the document includes
the following change:
 Updated section 11.5.1 Application Server, Network Server for EV 48848.

the following changes:
 Updated section 11.5.2 Element Management System for EV 65145.
 Added section 21.14.1 importdb.pl on Element Management System for EV 81954.
 Updated section 12 Disaster Recovery Strategy for EV 87423.
 Added section 11.1.3 TimesTen Replication Port Number.

the following change:
 Added section 17 Change BCCT Listening Ports for EV 62498.

the following changes:
 Made minor updates.
 Removed references to CMS.

Release 14.0, Version 15 is a maintenance version of this document. The following
change was made:
 Added the Profile Server information.
change was made:
 Added a description of all the input channels that are used in the BroadWorks
DebugConfig.xml files.
changes were made:
 Added section 10.5.7 TimesTen Connection Optimization for EV 60233.

©
 Added section 12.2.3.4 Xtended Services Platform Replacement.
changes were made:
 Added section 9.1.1 Configure Basic Maintenance Tasks for EV 59029.
 Updated section 5.1.7 Media Server for EV 59375. Note that this section has
subsequently been deleted from this document.
 Modified section 9 Scheduled Maintenance Tasks to add CDS and EMS daily
DbMaint task; also added section 21.12 getDbSize.pl for EV 53221.
 Created section on Optimization TimesTen Checkpoint for EV 59930.
change was made:
 Updated BEA Java GC logs information.
change was made:
 A procedure describing the process for importing TimesTen datastores between
different operating systems was added.
changes were made:
 Clarified the list of processes on the CMS.
 Added logging information for the CMS.
changes were made:
 Updated section 20.9 config-network to add a note for HTTP Alias CLI for EV 38723.
 Updated section 9.1.3 Restore Auto-Backup to indicate that the backup restore must
be executed as root when done in the system root directory for EV 50841.
Release 14.0, Version 6 is a maintenance version of this document. The following change
was made:
 Corrected log file named for 24-hour use for EV 49826.
changes were made to provide information concerning the use of BEA WebLogic with
BroadWorks.
 Provided new information on new and modified log files.
 Added licensing information for BEA.
 Added information on network configuration.
changes were made to the document:
 Config-network: This tool was migrated to a Web Server configuration-only tool.
The documentation was updated accordingly for the command, as well as for the
procedure used to change the IP address for a server.
Previous changes to the document include the following:
 Added new section for EV 37043, section 21.28 Software Manager Utilities.

©
 Changed note in section 8.1.1 Display Full Details about Active Software Version and
added text about -showPatch option in section 21.5 bwshowver for EV 37037.
 Added warning to section 21.23 stopbw for EV 39877.
 Added disaster recovery strategies, and minor textual updates to correct EV 34678.
 Added CommonPersistency input channel parameters.
2.8 Changes from Release 13.0

Release 13.0, Version 5 includes a new section listing all the key processes running on
BroadWorks.
changes were made to the document:
 Change IP Address: Updated procedure to change an IP address for a BroadWorks
server.
 ASDump: Clarified that the administrator must enter a HostingNe when running the
command. Specified a temporary location to store the output of the ASDump.
 Auto Backup: Updated procedure to show how to extract files from a backup image.
 Logging: Updated the BroadWorks debugging section. Added basic rules for
increasing the total number of log files for the Application Server Provisioning Server
(PS) and Execution Server (XS).
 Maintenance Tasks: Introduced procedure to add more memory for a BroadWorks
server.
 TimesTen: Added a procedure to perform a consolidation of the store in use for the
database by compacting it.

The following sections were added or modified since Release 12.0:
 Logging: Added a section about the BroadWorks bwauth.log file. Updated the
BroadWorks logging section.
 Linux: Generalized Solaris information and added Linux-specific sections when
required.
 Maintenance Tools: Added a description for: check_dbpages.pl
 Servers Address Management: Added a procedure to change the address of the
CDS, EMS, and Web Server.
 EMS Redundancy: Existing methods and procedures were updated and new ones
were added to cover for the EMS redundancy.
 Auto Backups: Added the P option to the tar xvf command so original full path is
restored on Linux.
 System Monitoring: Added sections describing which performance measurements to
monitor.
 BroadWorks Processes: The list of all BroadWorks processes per server type.


©
 Maintenance Tasks: Added definitions for three new maintenance tasks for the tech-
support tool.
 bwTestDaemon.pl: New command was added.
 Database: Added section “Updating Database Paging Information” to the document.

 Maintenance Recommendations: This section was added for Release 11.0.
 New tools as follows:
− Added description for syncheck_basic.pl, resizeDSN, and bwCPUMon.
− Renamed syncheck.pl to syncheck_full.pl.
 Maintenance Tasks: Added definitions for three new maintenance tasks:
bwCPUMon, regAudit, and syncheck.
 Server Management: A description for the server maintenance state was added to
the document.

©
3 Maintenance Recommendations
An operator should always use the following rules to maintain a BroadWorks server:
 Always use maintenance scripts and procedures provided by BroadSoft.
 Always refer to BroadWorks support personnel before installing any third-party
software that has not been distributed with the BroadWorks application.
 When scripts are used to maintain a server, you should always keep the /tmp
directory as empty as possible because the /tmp is shared with the /swap partition on
the server.
− Filling up the /tmp partition could cause a server’s performance to significantly
decline and may even cause it to stop functioning.
− It is recommended to use a directory other than /tmp to put temporary files and to
erase them at the end of a maintenance task.

©
4 BroadWorks Processes
All BroadWorks servers are composed of many processes. These processes can be
associated with the following categories:
 Application – The application processes are those that provide the BroadWorks
functionality, for example, the ExecutionAndProvisioning process on the Application
Server.
 Platform – The platform processes are those that provide external integration or
addition.
The following table provides a list of applications and their corresponding processes
installed with BroadWorks and it indicates which processes go with the different types
of servers. Note that you can find the expansion of abbreviations for server names in the
Abbreviations for Server Names table, which follows.
Application Process Servers
ExecutionAnd execution AS
Provisioning
provisioning
remotexla
WebContainer apache AS, NS, EMS, Xsp, PS,

UMS, USS
tomcat
Open Client Server OCS AS, EMS, Xsp
FlashPolicy flashPolicy AS, Xsp
AccessMediationServer AMSTurnServer AMS
AMSSignalingGateway AMS
keepalived AMS
LogServer LogServer MS, AMS, SCF, UMS,

USS, WRS
EMSBackEnd emsBackEnd EMS
MediaStreaming msfe MS
NSExecutionAndProvisioning nsExecution NS
nsProvisioning
remotexla
DeviceManagementTFTP DMTftp Xsp
RatingFunction RatingFunction Xsp
UC-Connect UC-Connect Xsp
Provisioning provisioning PS
subscriberAgent
Execution Dataless execution XS
subscriberAgent
DbManagement DbManagement DBS

©
Application Process Servers
DBSObserver dbsObserver PS
CCReportingDBManagement ccReportingDB PS
Management
EnhancedCallLogsDB enhancedCallLogsDB PS
Management Management
SCF Processor scfProcessor SCF
SS7ProtocolStack emSS7CAP SCF
emSS7MAP
emSS7Platform
ss7CAP
ss7CAPBridge
ss7MAP
ss7MAPBridge
ss7OAMBridge
ss7Platform
IMP dbConnector UMS
imp
muc
uss uss USS
WebRTCGateway gwcore WRS
ipcfe
stun
Configuration Agent BroadWorks All

(core platform application) Configuration Agent
SNMP Agent BroadWorks All

(core platform application) SNMP Agent
Software Manager BroadWorks All

(core platform application) Software Manager
Abbreviations for BroadWorks Server Names

The abbreviations for server names used in this guide are listed in the following table.
Abbreviation Server Name
AMS Access Mediation Server
AS Application Server
DBS Database Server
EMS Element Management System
MS Media Server

©
Abbreviation Server Name
NS Network Server
PS Profile Server
SCF Service Control Function Server
UMS Messaging Server
USS Sharing Server
WRS WebRTC Server
XS Execution Server
Xsp Xtended Services Platform
All application processes are monitored through the BroadWorks process monitor. If the
monitored process dies, the process monitor automatically restarts it. The healthmon
monitoring task validates that all deployed applications have all their processes running
and notifies the operator through an alarm if one of the processes listed above is not
present for a specific server. To view the list of BroadWorks application processes
running on a server, an operator can issue the showrun command from a UNIX prompt.
Example
bwadmin@mtl64lin12.mtl.broadsoft.com$ showrun
Currently running BroadWorks Application processes:
OCS process monitor (pid=12577)
OCS (pid=12656)
apache process monitor (pid=12499)
apache (pid=12644)
execution process monitor (pid=12426)
execution (pid=12957)
flashPolicy process monitor (pid=12547)
flashPolicy (pid=12585)
provisioning process monitor (pid=12453)
provisioning (pid=12992)
remotexla process monitor (pid=12404)
remotexla (pid=12430)
tomcat process monitor (pid=12476)
tomcat (pid=12513)
Currently running BroadWorks Platform processes:
BroadWorks Configuration Agent process monitor (pid=8656)

BroadWorks Configuration Agent (pid=8686)
BroadWorks Software Manager (pid=11894)
BroadWorks SNMP Agent process monitor (pid=7117)
BroadWorks SNMP Agent (pid=7147)

©
5 Logs
BroadWorks supports full logging capability. Certain logs are system-controlled, whereas
others are on all the time, and other logs such as debug logging, can be modified to
provide more information when required. BroadWorks application-related log files that
reside under /var/broadworks/logs include the following.
Process/ Server Log File Location

Web Application Type
execution AS, XS Process Monitor Logs: appserver/executioncontainerpmdXX.log

Debug Logs: appserver/XSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: appserver/XSOutputXX.log
provisioning AS, PS AS:

Process Monitor Logs: appserver/provisioningcontainerpmdXX.log
Debug Logs: appserver/PSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: appserver/PSOutputXX.log
Audit Debug Logs: appserver/AuditLogyyyy.mm.dd-hh.mm.ss.txt
PS:
Process Monitor Logs: provisioning/provisioningcontainerpmdXX.log
Debug Logs: provisioning/PSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: provisioning/PSOutputXX.log
Audit Debug Logs: provisioning/AuditLogyyyy.mm.dd-hh.mm.ss.txt
Remotexla AS, NS Process Monitor Logs: remotexla/apachecontainerpmdXX.log

Stdout Logs: remotexla/RemoteXlaOutputXX.log
CommPilot (webApp) AS Debug Logs: appserver/CommPilotLogyyyy.mm.dd-hh.mm.ss.txt

Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.
DeviceManagement AS Debug Logs: appserver/DeviceManagementFilesLogyyyy.mm.dd-

Files (webApp) hh.mm.ss.txt
JWSFiles (webApp) AS Debug Logs: appserver/JWSFilesLogyyyy.mm.dd-hh.mm.ss.txt
MediaFiles (webApp) AS Debug Logs: appserver/MediaFilesLogyyyy.mm.dd-hh.mm.ss.txt
OCIFiles (webApp) AS Debug Logs: appserver/OCIFilesLogyyyy.mm.dd-hh.mm.ss.txt

Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.
OCIOverSoap AS Debug Logs: appserver/OciOverSoapLogyyyy.mm.dd-hh.mm.ss.txt

(webApp) Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.
apache AS, NS, Process Monitor Logs: apache/remotexlacontainerpmdXX.log

EMS, Xsp, Stdout Logs: apache/ApacheOutputXX.log
PS, UMS
tomcat AS, NS, Process Monitor Logs: tomcat/tomcatcontainerpmdXX.log

EMS, Xsp, Stdout Logs: tomcat/TomcatOutputXX.log
PS, UMS
OCS AS, Xsp, Process Monitor Logs: openclientserver/openclientserverOutputXX.log

EMS Debug Logs: openclientserver/OCSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: appserver/openclientserverOutputXX.log
flashPolicy AS Process Monitor Logs: flashPolicy/flashPolicycontainerpmdXX.log

Xsp Stdout Logs: flashPolicy/flashpolicydOutputXX.log
AMSTurnServer AMS Process Monitor Logs: ams/AMSTurnServercontainerpmdXX.log

Debug Logs: ams/TSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ams/AMSTSOutputXX.log

©
AMSSignalingGate AMS Process Monitor Logs: ams/AMSSignalingGatewaycontainerpmdXX.log

way Debug Logs: ams/AMSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ams/AMSOutputXX.log
keepalived AMS Process Monitor Logs: keepalived/keepalivedcontainerpmdXX.log

Stdout Logs: keepalived/keepalivedOutputXX.log
LogServer AMS, MS, Process Monitor Logs: logserver/LogServercontainerpmdXX.log

UMS, Debug Logs: logserver/logServerLogyyyy.mm.dd-hh.mm.ss.txt
USS, WRS Stdout Logs: logserver/LogServerOutputXX.log
emsBackEnd EMS Process Monitor Logs: emsBackEnd/emsBackEndcontainerpmdXX.log

Debug Logs: emsBackEnd/EmsBeLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: emsBackEnd/emsbeOutputXX.log
The /var/broadworks/logs/emsBackEnd directory also contains a set of
detailed debug logs the functionality provided by the EMS.
emsFrontEnd EMS Debug Logs: emsFrontEnd/EmsFeLogyyyy.mm.dd-hh.mm.ss.txt

(webApp) The /var/broadworks/logs/emsFrontEnd directory also contains a set of
detailed debug logs the functionality provided by the EMS.
LogRepository EMS, PS Debug Logs: logrepos/LogRepositoryLogyyyy.mm.dd-hh.mm.ss.txt

(webApp)
Msfe MS Process Monitor Logs: mediaserver0/msfecontainerpmdXX.log

Debug Logs: mediaserver0/msfeyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: mediaserver0/msfeOutputXX.log
nsExecution NS Process Monitor Logs: routingserver/nsExecutioncontainerpmdXX.log

Debug Logs: routingserver/NSXSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: routingserver/XSOutputXX.log
nsProvisioning NS Process Monitor Logs: routingserver/nsProvisioningcontainerpmdXX.log

Debug Logs: routingserver/NSPSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: routingserver/PSOutputXX.log
NSPortal (webApp) NS Debug Logs: nsportal/NSPortalLogyyyy.mm.dd-hh.mm.ss.txt
DMTftp Xsp Process Monitor Logs: tftp/DMTftpcontainerpmdXX.log

Debug Logs: tftp/tftpLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: tftp/tftpLogXX.log
RatingFunction Xsp Process Monitor Logs: RatingFunction/ RatingFunctioncontainerpmdXX.log

Debug Logs: RatingFunction/RatingFunctionLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: RatingFunction/RatingFunctionXX.log
UC-Connect Xsp Process Monitor Logs: UC-Connect/UC-ConnectcontainerpmdXX.log

Debug Logs: UC-Connect/UC-ConnectLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: UC-Connect/UC-ConnectOutputXX.log
subscriberAgent XS, PS Process Monitor Logs: subscriber/subscriberAgentcontainerpmd XX.log

Debug Logs: subscriber/SubscriberAgentyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: subscriber/subscriberOutput XX.log
DbManagement DBS Process Monitor Logs: dbm/DbManagementcontainerpmd XX.log

Debug Logs: dbm/DbManagementyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: dbm/DBMLogOutput XX.log
dbsObserver PS Process Monitor Logs: dbsObserver/dbsObservercontainerpmd XX.log

Debug Logs: dbsObserver/dbsObserveryyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: dbsObserver/dbsObserverOutput XX.log
CCReporting PS Debug Logs: profileserver/CCFileReposLogyyyy.mm.dd-hh.mm.ss.txt

Repository (webApp)
CCReporting PS Debug Logs: profileserver/CCReportingLogyyyy.mm.dd-hh.mm.ss.txt

(webApp)

©
ccReportingDB PS Process Monitor Logs:

Management profileserver/enhancedCallLogsDBManagementcontainerpmdXX.log
Debug Logs: profileserver/EnhancedCallLogsDBLogyyyy.mm.dd-
hh.mm.ss.txt
Stdout Logs: profileserver/EnhancedCallLogsDBLogXX.log
enhancedCallLogsDB PS Process Monitor Logs:

Management profileserver/ccReportingDBManagementcontainerpmdXX.log
Debug Logs: profileserver/CCReportingDBLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: profileserver/CCReportingDBLogXX.log
scfProcessor SCF Process Monitor Logs: scf/scfProcessorcontainerpmdXX.log

Debug Logs: scf/SCFProcessLog yyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: scf/scfprocessorOutputXX.log
emSS7CAP SCF Process Monitor Logs:

emSS7CAP/emSS7CAPProcessorcontainerpmdXX.log
Stdout Logs: emSS7CAP/emSS7CAP XX.log
emSS7MAP SCF Process Monitor Logs:

emSS7MAP/emSS7MAPProcessorcontainerpmdXX.log
Debug Logs:
emSS7MAP/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: emSS7MAP/emSS7MAP XX.log
emSS7Platform SCF Process Monitor Logs:

emSS7Platform/emSS7PlatformcontainerpmdXX.log
Debug Logs:
emSS7Platform/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Debug Logs: emSS7Platform/ss7p_events_xxxxx_MMM_dd_hh.mm.ss.log
Debug Logs: emSS7Platform/LOGS_EM-xxxxx-MMM_dd_hh.mm.ss
Stdout Logs: emSS7Platform/emSS7PlatformXX.log
ss7Platform SCF Process Monitor Logs: ss7Platform/ss7PlatformcontainerpmdXX.log

Debug Logs:
ss7Platform/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Debug Logs: ss7Platform/LOGS_FEP_Inst_x-xxxxx-MMM_dd_hh.mm.ss
Stdout Logs: ss7Platform/ss7PlatformXX.log
ss7CAP SCF Process Monitor Logs: ss7CAP/ss7CAPcontainerpmdXX.log

Debug Logs:
ss7CAP/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7CAP/ss7CAPXX.log
ss7MAP SCF Process Monitor Logs: ss7MAP/ss7MAPcontainerpmdXX.log

Debug Logs:
ss7MAP/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7MAP/ss7MAPXX.log
ss7CAPBridge SCF Process Monitor Logs: ss7Bridge/ ss7CAPBridgecontainerpmdXX.log

Debug Logs: ss7Bridge/ SS7CAPyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ss7Bridge/ss7CAPBridgeXX.log
ss7MAPBridge SCF Process Monitor Logs: ss7Bridge/ ss7MAPBridgecontainerpmdXX.log

Debug Logs: ss7Bridge/ SS7MAPyyyy.mm.dd-hh.mm.ss.txt
Debug Logs:
ss7Bridge/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7Bridge/ss7MAPBridgeXX.log
ss7OAMBridge SCF Process Monitor Logs: ss7Bridge/ss7OAMBridgecontainerpmdXX.log

Debug Logs: ss7Bridge/SS7OAMyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ss7Bridge/ss7OAMBridgeXX.log
dbConnector UMS Process Monitor Logs: ums/dbConnectorcontainerpmdXX.log

Debug Logs: ums/dbConnectorLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ums/DBConnectorOutputXX.log

©
imp UMS Process Monitor Logs: ums/impcontainerpmdXX.log

Debug Logs: ums/IMPLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ums/IMPOutputXX.log
muc UMS Process Monitor Logs: ums/muccontainerpmdXX.log

Debug Logs: ums/MUCLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: ums/MUCOutputXX.log
provisioningAdapter UMS Debug Logs: ums/ProvisioningAdapterLogyyyy.mm.dd-hh.mm.ss.txt
uss USS Process Monitor Logs: uss/usscontainerpmdXX.log

Debug Logs: uss/USSLogyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: uss/ussOutputXX.log
gwcore WRS Process Monitor Logs: webrtcserver/gwcorecontainerpmdXX.log

Debug Logs: webrtcserver/GWcoreyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: webrtcserver/gwcoreXX.log
ipcfe WRS Process Monitor Logs: webrtcserver/ipcfecontainerpmdXX.log

Debug Logs: webrtcserver/IPCFEyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: webrtcserver/ipcfeXX.log
BroadWorks All Process Monitor Logs: config/configplatformpmdXX.log

Configuration Agent Debug Logs: config/configdyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: config/configdOutputXX.log
BroadWorks SNMP All Process Monitor Logs: snmp/snmpplatformpmdXX.log

Agent Debug Logs: snmp/snmpyyyy.mm.dd-hh.mm.ss.txt
Stdout Logs: snmp/snmpXX.log
BroadWorks Software All Process Monitor Logs: swmanager/snmpplatformpmdXX.log

Manager Debug Logs: swmanager/swmanagerActivities yyyy.mm.dd-hh.mm.ss.txt
The /var/broadworks/logs/swmanager directory also contains patching
activity logs.
The following table lists logs related to clustered server configuration and maintenance
activities. Only applicable servers are listed.
Log File Location
Event Type
AS NS EMS PS SCF UMS
File replication-related rsync/ rsync/ rsync/ rsync/ rsync/ rsync/

events frepXX log frepXX.log frepXX.log frepXX.log frepXX.log frepXX.log
For all servers, the server state management-related logs are located in the
maintenance/MoExtensionsXXXXXX.log file.
5.1 BroadWorks Debug Logs

Debug logs leave a trace of BroadWorks events in chronological order. Debug logs
provide a trace of all internal and external events related to all calls and sessions. By
default, logging is set at the highest severity level (Debug), with only the Session Initiation
Protocol (SIP) and Media Gateway Control Protocol (MGCP) input channels active.
WARNING: Increasing the logging level affects server performance. Therefore, turn on input
channels with caution. By default, only the protocol input channels are on.

©
5.1.1 Log Severity
The severity of the logs is set on the input channels. Set either all or individual input
channels. The following example shows the severity level settings for the CallP input
channel:
 Warning – General warning messages
 Info – General informational messages
 Notice – Anomalous but gracefully handled behavior
 FieldDebug – Highest level of debugging to be used on a production system
 Debug – Highest level of debugging – all messages
The severity should be left at the default FieldDebug level
5.1.2 Log Input Channels

BroadWorks sends logs to a number of input channels, each related to a particular sub-
system. For example, the CommonPersistency channel relates to the TimesTen
database. Enable a channel to send logs to a file. Disable a channel to ignore logs.
When troubleshooting, enable channels to better understand what may be the cause of a
failure. On all BroadWorks servers, the logging is controlled through the CLI. Each
application has a logging context under its application context. For instance, the
Execution (XS) logging is controlled under the AS_CLI/Applications/ExecutionAnd
Provisioning/XS/Logging> CLI level.
Within the CLI, each application context is listed under the application context. From each
application context, a logging context is available (where applicable) to configure the
logging for the application.
To enable logs for a specific input channel, change the Enabled property from “false” to
“true”.
However, one component still makes use of an eXtensible Markup Language (XML) log
configuration file, which is the following:
 Software Manager: The file /usr/local/broadworks/swmanager/conf/patchLogger.conf
controls the log settings.
WARNING: Be very careful when editing the logging configuration on the Application Server or
Network Server.
5.1.2.1 Accounting Input Channel

Channel Name(s) Accounting
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel that can be used to dump all accounting events in
to the log file.
Special None
Configuration

©
5.1.2.2 Apache Commons Logging Input Channels
Channel Name(s) HttpClient, DefaultCommonsLogging, Soap
Applies To ExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump the logs generated
when using the Apache HttpClient. The Apache HttpClient is
used with the external file system integration in the Application
Server Provisioning Server process.
Special None
Configuration
5.1.2.3 Audit Log Input Channel

Channel Name(s) Audit
Applies To ExecutionAndProvisioning (PS and XS) application,
Provisioning application
Description Input channel that can be used to:
 For the Application Server, dump the OCI events to the
AuditLog file.
 For the Network Server, dump the OCI and Network
Server legacy audit logs to the AuditLog file.
Special The Audit input channel has the following additional controls:
Configuration
 GetRequests – Indicates whether OCI get requests are
logged.
 AccountInfo – Indicates whether the user ID and
authorization level of the account executing the request is
included.
 Verbose – Indicates whether the xml string of the OCI
request is written.
5.1.2.4 Authentication Input Channel

Channel Name(s) Authentication
Applies To USS application
Description Input channel used to report logs related to authentication
activities for the USS application on the Sharing Server.
Special None
Configuration

©
5.1.2.5 BroadWorks Common Communication Transport Input Channels
Channel Name(s) BroadsoftCommonCommunicationTransport,
BroadsoftCommonCommunicationTransportKeepAlive
NSExecutionAndProvisioning (NSPS and NSXS) application,
OpenClientServer application, SNMPDebugConfig.xml,
NSPortal application, patchLogger.conf, ExecutionDataless
application, Provisioning application
Description The BCCT transport mechanism is used to transport various
protocols for BroadWorks inter-process communications. The
BroadsoftCommonCommunicationTransport channel can be
enabled to view the messages exchanged between servers.
The BroadsoftCommonCommunicationTransportKeepAlive can
be enabled to view the BCCT Keep Alive messages exchanged
between servers.
Special None
Configuration
5.1.2.6 Call Center Reporting Input Channel

Channel Name(s) CCRChannel, DatabaseChannel, CommunicationChannel
Applies To CallCenterReporting application
Description Input channel used to log internal Call Center Reporting (CCR)
processing debugging information. The DatabaseChannel can
be used to view all access to the database, while the
CommunicationChannel captures all inter-process
communication events for the CCR.
Special None
Configuration
5.1.2.7 CallP Input Channel

Channel Name(s) CallP
application
Description Input channel used on the Application Server to output internal
call processing events.
Special None
Configuration
5.1.2.8 CallP Application Protocol Input Channel

Channel Name(s) CAP
application
Description Input channel used to output all incoming and outgoing CAP
messages.

©
Special None
Configuration
5.1.2.9 CommonPersistency Input Channel

Channel Name(s) CommonPersistency
NSExecutionAndProvisioning (NSPS and NSXS) application,
Provisioning application, ExecutionDataless application
Description BroadWorks database abstraction layer-specific logs. The logs
exposed by this channel represent internal information captured
by the abstraction layer.
Special The CommonPersistency input channel has additional
Configuration parameters, which are used to monitor the database
subsystem. These parameters are omitted by default and
should only be added when requested or authorized by
BroadSoft personnel.
The parameters are as follows:
 UpdateTimeThreshold – Specifies a time in milliseconds.
If a database update takes longer than the specified
threshold, a log and SNMP trap are output.
 QueryTimeThreshold – Specifies a time in milliseconds. If
a database query takes longer than the specified
threshold, a log and SNMP trap are output.
 UpdateRowsThreshold – Specifies a number of rows. If a
database update modifies more than this number of rows,
a log, and SNMP trap are output.
 QueryRowsThreshold – Specifies a number of rows. If a
database query returns, more than this number of rows,
then a log and an SNMP trap are output.
WARNING: Setting these parameters with incorrect values can lead to severe performance
degradations. Do NOT modify these settings unless requested by BroadSoft personnel.
5.1.2.10 BWCommunicationUtility Input Channel

Channel Name(s) BWCommunicationUtility
Applies To FileRepos application
Description Input channel used to report logs related to the
BWCommunicationUtility running in Tomcat on the Profile
Server.
Special None
Configuration

©
5.1.2.11 DbConnector Input Channel
Channel Name(s) DbConnector
Applies To IMP application
Description Input channel used to report logs related to the DbConnector
process running in the IMP application on the Messaging
Server.
Special None
Configuration
5.1.2.12 Event Notification Input Channel

Channel Name(s) EventNotification
application
Description Input channel used to output logs related to the Application
Server event notification framework. The event notification
framework is used for subscription-based services, for example,
Busy Lamp Field and Automatic Callback.
Special None
Configuration
5.1.2.13 File Repository Input Channel

Channel Name(s) FileRepos
Applies To FileRepos application.
Description Input channel used to report logs related to the file repository
web application running in Tomcat on the Profile Server.
Special None
Configuration
5.1.2.14 File System Input Channel

Channel Name(s) FileSystem
Applies To Configuration done through BroadWorks Configuration Agent
Description Input channel used to report logs related to the external file
system integration.
Special None
Configuration

©
5.1.2.15 Generic Input Channel
Channel Name(s) Generic
Applies To All applications, SNMPDebugConfig.xml, patchLogger.conf
Description Input channel used to report miscellaneous logs from the
process.
Special None
Configuration
5.1.2.16 IMP Input Channel

Channel Name(s) IMP
Applies To IMP application
Description Input channel used to report logs related to the imp process
running in the IMP application on the Messaging Server.
Special None
Configuration
5.1.2.17 Sh Interface Input Channel

Channel Name(s) ShInterface
Applies To ExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump all events related to the
Sh (IMS mode only) interface integration.
Special None
Configuration
5.1.2.18 SynchAPI Input Channel

Channel Name(s) SynchAPI
Applies To NSExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump all incoming and
outgoing SyncAPI messages.
Special None
Configuration

©
5.1.2.19 LocationAPI Input Channel
Channel Name(s) LocationAPI
Applies To NSPortal application
Description Input channel used to output all incoming and outgoing
LocationAPI (Network Server Web Server application)
messages.
Special None
Configuration
5.1.2.20 Media Gateway Controller Protocol Input Channel

Channel Name(s) MGCP
application
Description Input channel used to output all incoming and outgoing MGCP
messages.
Special None
Configuration
5.1.2.21 IMP Input Channel

Channel Name(s) MUC
Applies To MUC application
Description Input channel used to report logs related to the muc process
running in the IMP application on the Messaging Server.
Special None
Configuration
5.1.2.22 Name Service Input Channel

Channel Name(s) NameService
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application
Description Input channel used to output all naming service lookup
messages along with additional internal debugging information.
Special None
Configuration

©
5.1.2.23 Network Resource Service Input Channel
Channel Name(s) NRSLog
Description Input channel used to report Network Resource Service (NRS)-
related logs. NRS is the underlying framework that is used by
Application Server Redundancy (ASR), the BroadWorks
Redundancy Protocol used to exchange migration information
from the Application Server to the Network Server and from the
Application Server to the Application Server. Enabling the
channel allows the ability to view incoming and outgoing ASR
messages.
Special None
Configuration
5.1.2.24 Open Client Server Input Channel

Channel Name(s) OpenClientServer
Applies To OpenClientServer application
Description Input channel used to log internal Open Client Server process
debugging information.
Special None
Configuration
5.1.2.25 Overload Control Validation Input Channel

Channel Name(s) Overload
Description Input channel used to report SIP and MGCP-related overload
control events.
Special None
Configuration
5.1.2.26 Provisioning Adapter Input Channel

Channel Name(s) PAdapter
Applies To PAdapter web application
Description Input channel used to report logs related to the Provisioning
Adapter web application running in tomcat on the Messaging
Server.
Special None
Configuration

©
5.1.2.27 Patch Tool Input Channels
Channel Name(s) PatchTool, PatchToolBcctTraffic, PatchToolPersistence
Applies To patchLogger.conf
Description Input channel used to log Software Manager/patching
debugging information.
Special None
Configuration
5.1.2.28 PortalAPI Input Channel

Channel Name(s) PortalAPI
Description Input channel that can be used to dump all incoming and
outgoing PortalAPI messages.
Special None
Configuration
5.1.2.29 Provisioning Validation Input Channel

Channel Name(s) ProvisioningValidation
Applies To ExecutionAndProvisioning (XS and PS) application,
Description Input channel used by the Application Server Provisioning
Server (PS) to the Execution Server (XS) Provisioning
Validation Protocol. It can be used to dump all provisioning
validation-related messages to the log files.
Special None
Configuration
5.1.2.30 Room Control Input Channel

Channel Name(s) RoomControl
Description Input channel used to report logs related to Room Control
Special None
Configuration
5.1.2.31 Room Manager Input Channel

Channel Name(s) RoomManager
Description Input channel used to report logs related to Room Manager
Special None
Configuration

©
5.1.2.32 Service Control Proxy Input Channel
Channel Name(s) ServiceControlProxy
Description Input channel used to output all incoming and outgoing Service
Control Proxy (Network Server Web Server application)
messages.
Special None
Configuration
5.1.2.33 Session Initiated Protocol Input Channel

Channel Name(s) SIP
ExecutionDataless application, WebRTCGatewayapplication.
Description Input channel used to output all incoming and outgoing SIP
messages.
Special None
Configuration
5.1.2.34 Session Initiated Protocol Media Input Channel

Channel Name(s) SIPMedia
application
Description Input channel used to output all incoming and outgoing SIP
messages exchanged between the Application Server and the
Media Server.
Special None
Configuration
5.1.2.35 Sharing Control Input Channel

Channel Name(s) SharingControl
Description Input channel used to report logs related to Sharing Control
Special None
Configuration
5.1.2.36 Sharing Data Input Channel

Channel Name(s) SharingData
Description Input channel used to report logs related to Sharing Data

©
Special None
Configuration
5.1.2.37 Short Message Peer-to-Peer Input Channel

Channel Name(s) SMPP
application
Description Input channel used to output all incoming and outgoing Short
Message Peer-to-Peer Protocol (SMPP)-related messages.
Special None
Configuration
5.1.2.38 Simple Management Application Protocol Input Channel

Channel Name(s) SMAP
Applies To ExecutionAndProvisioning (XS and PS) application,
NSExecutionAndProvisioning (NSXS and NSPS) application,
OpenClientServer application, SNMPDebugConfig.xml,
NSPortal application, ExecutionDataless application,
Description Input channel used to capture the alarm and performance
measurement (PM)-related information exchanged between the
BroadWorks processes and the SNMP agent.
Special None
Configuration
5.1.2.39 Simple Message Desk Interface Input Channel

Channel Name(s) SMDI
application
Description Input channel used to output all incoming and outgoing
simplified message desk interface (SMDI) messages.
Special None
Configuration
5.1.2.40 Simple Network Management Protocol Input Channel

Channel Name(s) SNMP
Applies To SNMPDebugConfig.xml
Description Input channel used to capture SNMP agent internal logs.
Special None
Configuration

©
5.1.2.41 ServiceOS Input Channel
Channel Name(s) ServiceOS
Description The ServiceOS is an internal framework commonly used by the
Application Server Execution Server and Network Server
Execution Server. Logs on this channel represent internal
framework events.
Special None
Configuration
5.1.2.42 Timer Input Channel

Channel Name(s) Timer
application
Description The Application Server Execution Server uses internal timers.
Logs related to timers waking up and processing problems are
reported through this channel.
Special None
Configuration
5.1.2.43 Subscriber Subsystem Input Channel

Channel Name(s) SubscriberSubsystem
Applies To ExecutionDataless application, Provisioning application
Description This input channel is used by the Subscriber Agent.
Special None
Configuration
5.1.2.44 SCF Processor Input Channel

Channel Name(s) SCFProcessor
Applies To SCFProcessor application
Description This input channel is used by the SCF processor.
Special None
Configuration
5.1.2.45 SS7 CAP Input Channel

Channel Name(s) SS7Cap
Applies To SCFProcessor application, SS7ProtocolStack application
Description This input channel is used by the SS7 CAP.
Special None
Configuration

©
5.1.2.46 SS7 MAP Input Channel
Channel Name(s) SS7Map
Applies To SCFProcessor application, SS7ProtocolStack application
Description This input channel is used by the SS7 MAP.
Special None
Configuration
5.1.2.47 SS7 OAM Input Channel

Channel Name(s) SS7oam
Applies To SS7ProtocolStack application
Description This input channel is used by the SS7 OAM.
Special None
Configuration
5.1.2.48 Web Socket Input Channel

Channel Name(s) webSocket
Applies To WebRTCGateway application
Description Input channel used to report logs related to Web Socket
activities for the WebRTCGateway application on the WebRTC
Server.
Special None
Configuration
5.1.2.49 XML IPC Input Channel

Channel Name(s) XML-IPC
Applies To WebRTCGateway application
Description Input channel used to report logs related to interprocess
communication (IPC) activities between the processes of the
WebRTCGateway application on the WebRTC Server.
Special None
Configuration
5.1.3 Input Channel Attributes Channels

The attributes for the input channels are configured through the CLI under each
application logging context. The following Profile Server example shows the logging
context commands and subcontexts.
PS_CLI/Applications/BroadworksFileRepos/Logging> ?
0) get : show Logging-related attributes
1) set : modify Logging-related attributes
2) InputChannels : go to level InputChannels
3) OutputChannels : go to level OutputChannels
h (help), e (exit), q (quit), r (read), w (write), t (tree),

©
c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)
PS_CLI/Applications/BroadworksFileRepos/Logging> InputChannels
PS_CLI/Applications/BroadworksFileRepos/Logging/InputChannels> ?
0) get : show InputChannels-related attributes
1) set : modify InputChannels-related attributes
2) clear : clear InputChannels-related attributes

(repeat)
PS_CLI/Applications/BroadworksFileRepos/Logging/InputChannels> get
Name Enabled Severity
===========================================
BWCommunicationUtility true
FileRepos true
Generic true
3 entries found.
5.1.4 Enable or Disable BroadWorks Debug Logging

The debug logging is enabled and disabled from the CLI. Under the application context,
the enabled attribute can be changed and the change takes effect automatically.
NS_CLI/Applications/NSExecutionAndProvisioning/XS/Logging/InputChannels>
get
Name Enabled Severity
===================================================================
Generic true Debug
Debug false
ServiceOS true
NRSLog true
CommonPersistency true Info
BroadsoftCommonCommunicationTransport true Info
BroadsoftCommonCommunicationTransportKeepAlive false Info
CallP true
SMAP false
Sip true
Timer true
Overload true
12 entries found.
For the SNMP agent, the script /user/local/broadworks/bw_base/bin/bwdebugreload must

be used to reload any changes to the debug configuration file.
5.1.5 Output Channel attributes

The default amount of disk space used by the logs for the Application Server is 7 GB (1
GB for Provisioning Server logs, 6 GB for Execution Server logs). By changing the
OutputChannel “FileSizeMegs” and “NumberOfFiles”, you can control the amount of disk
space used by the Provisioning and Execution Server log files.
Attribute Definition
Name “File” to indicate to the logging framework to use the file output channel.

©
Attribute Definition
Directory Indicates where the log files are stored.
File Prefix The string that the log file starts with.
The size the file should grow to before it is closed and a new file is created.
File Size
The size is set in MB.
NumberOfFiles The amount of files to store before the oldest file is deleted.
Enabled Turns on the output channel.
It is recommended to augment the number of log files when the disk size permits it. This
makes sure that log files are kept for a longer period of time, which allows better
troubleshooting when required. The formula for defining the total number of Execution
Server and Provisioning Server logs is as follows:
 The Provisioning Server should never be more than 20% of the Execution Server
logs. The Execution Server generates many more logs per minute, so it reaches it
maximum much more quickly than the Provisioning Server does.
 The total of Execution Server and Provisioning Server logs should never be more than
33% of the total partition size on which the logs are residing. This makes sure that
other applications have available disk space for their logs and the logs would not
conflict with the Application Server database.
5.1.6 Log Rotation
WARNING: If usage of the partition in which /var/broadworks/logs/appserver grows to 100%,

the Application Server crashes. It is important that disk usage be monitored to make sure this
does not occur.
There are log files for each day of the month such as, Aspmd, Snaspmd, Snmpas,
Snmptraps.log, XSOutput##.log, and PSOutput##.log. The day of the month is appended
to the file name, and these files are stored until they are overwritten the next month. There
are also Application Server and Provisioning Server logs that are rotated by size and are
deleted when the configured number of files to store is reached. Delete unnecessary old
files (files not currently being written to).
5.1.7 Sample Debug Configuration File

All debug logging configurations are done through the CLI under the application context
(for example, AS_CLI/Application/CommPilot/Logging). However, there is still an
exception; the Software Manager still use a debug configuration file. The following is an
example of a debug configuration file for the Software Manager.

<Log Enabled="True" Priority="4">
<InputChannels Severity="Debug">
<InputChannel Name="SWManager" Enabled="True" Severity="Debug"/>
<InputChannel Name="PatchToolBcctTraffic" Enabled="True"/>
<InputChannel Name="Generic" Enabled="True" Severity="Debug"/>
<InputChannel Name="BroadsoftCommonCommunicationTransport"
Enabled="True" Severity="Debug"/>
<InputChannel Name="BroadsoftCommonCommunicationTransportKeepAlive"
Enabled="False" Severity="Info"/>
<InputChannel Name="PatchToolPersistence" Enabled="True"/>

©
<InputChannel Name="SmapInputChannel" Enabled="True"
Severity="Debug"/>
</InputChannels>
<OutputChannels>
<OutputChannel Name="Stdout" Enabled="False" />
<OutputChannel Name="File" Dir="/var/broadworks/logs/swmanager/"
FilePrefix="swmanagerActivities_" FileSizeMegs="30" NumberOfFiles="200"
Enabled="True" />
</OutputChannels>
</Log>
5.1.8 Platform Daemons Logging

The logging for the platform daemons is configured from the CLI at specific levels. The
same configuration parameters, as described in previous sections, are available for the
platform daemons logging. The CLI level for each platform daemon is as follows:
 BroadWorks Configuration Agent: CLI/System/ConfigAgent/Logging>

 BroadWorks SNMP Agent: CLI/Interface/SNMP/Logging>
5.2 BroadWorks UNIX Command Execution Logging

Every time a BroadWorks UNIX command is run, a syslog is generated and stored in a file
under /var/adm/bwauth.log. A log contains information such as who ran the command,
when it was executed, and the list of arguments that were passed to the command.
bwadmin@mtl64lin02$ cat bwauth.log | head -10
Jun 15 09:25:14 mtllinuxmisc04 BROADWORKS[3723]: uid=0(root) gid=0(root) groups=
0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) COMMAND=./bwuseradd --s
hell=/bin/ksh --role=BWORKS --passwd=bwadmin bwadmin
Jun 15 09:30:39 mtllinuxmisc04 BROADWORKS[3045]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/switch
-context.pl
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/dbsetu
p.sh AS_Rel_13.0_1.193
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/MoExte
nsions.pl -getMaintState
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/timest
en.pl -dsn appserver load
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/create
tables.sh
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/loadba
sedata.sh
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/bwPeriodMaint.sh

) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/bwgetflock.pl /u
sr/local/broadworks/bw_base/bin/bwPeriodMaint.sh bwAutoCleanup.sh bwAutoBackup.s
h
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/MoExtensions.pl
-getMaintState

©
5.3 Miscellaneous Logs
5.3.1 Solaris System Logs

The /var/adm/messages file contains all platform hardware, operating system, and kernel
logs, which are helpful to debug platform-related issues.
5.3.2 Linux System Logs

The /var/log/messages file contains all platform hardware, operating system, and kernel
logs, which are helpful to debug platform-related issues.
5.3.3 Enable TimesTen Logs

The TimesTen logs are enabled by default and they are controlled by the
/var/TimesTen/<ttVersions>/ttendaemon.options file.
TimesTen outputs logs through syslog to /var/adm/messages on Solaris or
/var/log/messages on Linux. By default, all user.info level of severity TimesTen logs is
directed to the messages file. This generates a large number of non-error-related logs.
Reduce the level of logging activity by commenting on the following non-error channel
from the /etc/syslog.conf file. Example:
user.info /var/adm/messages
To apply the changes, reset the syslogd daemon (/etc/rc2.d/ S74syslog script).
5.3.4 Apache Logs

The Apache error logs are rotated daily and are stored in /var/broadworks/logs/apache.
The name of each log is in the format error_log.MM.DD.YYYY.
5.3.5 Tomcat Logs

The Tomcat logs are rotated daily and are stored in /var/broadworks/logs/tomcat. The
name of each log is in the format of TomcatOutputDD.log, where DD is the day of the
month.
5.3.6 Oracle Logs

The Oracle database running on the Database Server exposes various logs files, each
associated with a functional component of the database. The description of the relevant
log files is documented in the BroadWorks Database Server Configuration Guide.

©
6 Fault and Performance Measurements
The BroadWorks Performance Measurements Interface Specification Guide and the

BroadWorks Fault and Alarm Interface Specification Guide contain detailed information
about faults and performance measurements for the system.
Simple Network Management Protocol (SNMP) v2c/v3 traps and get are supported.
6.1 Alarms
The Network Server, Application Server, Provisioning Server, Execution Server, and
Media Server provide alarms. The complete list of alarms can be found in the
BroadWorks Fault and Alarm Interface Specification Guide.
BroadWorks generated faults can be one of the following severities:
Alarm Level Action Necessary
Informational None
Low Check the affected system area; this alarm typically indicates a problem with
connectivity, and the redundancy of the system may be in jeopardy.
Medium Check the affected system area; this alarm typically indicates a problem with
connectivity, and the redundancy of the system may be in jeopardy.
High Contact BroadSoft immediately.
Critical Contact BroadSoft immediately.
6.1.1 Open System Alarms

Use this command to open a connection with the Simple Network Management Protocol
(SNMP) agent to obtain system alarms and event logs generated by BroadWorks.
6.1.1.1 Open Connection for Alarms

From the CLI Monitoring/Alarm level, type open [port]. Valid values include:
Field Type Valid Values Description
[port] Integer 1000 to 65535 Optional – The port used to
communicate with the SNMP agent.
(Port 30002 is recommended and is
the default value.)
Example:
Open
Using port 30002...
...Done

©
6.1.2 View System Alarms
Use this command to get and search for system alarms in the local CLI backlog. Use the
command alone to display all alarms in the local CLI backlog. Use the command with a
specified variable to display alarms that match the criteria.
6.1.2.1 Get and Search for System Alarms

From the CLI Monitoring/Alarm level, type get to display all alarms, or type
get [<attribute> <relation> <value>] [<back>] to specify which alarms to
display. Valid values include:
Variable Field Attribute Valid Values Description
<attribute> type Choice Notification The trap is for informational
purposes.
Alarm The trap is reporting an alarm.
Software Error The trap is reporting a software
error.
TrapName String 0 to 255 The trap name.
characters
Problem String 0 to 255 The text indicating details of the
characters problem.
Subcomp String 0 to 255 Software component within the
characters BroadWorks entity, reporting the
alarm: SIP, database, and so on.
Recommend String 0 to 255 The recommended action to take in
action characters response to this trap.
State Choice Off Alarm is off.
On Alarm is on.
Time String 0 to 255 The date and time at which the
characters alarm was generated.
Comp String 0 to 255 BroadWorks entity reporting the
characters alarm: Application Server, Media
Server, or Network Server.
Sysname String 0 to 255 The host name of the system
characters running the BroadWorks software.
TrapSeverity Choice Informational Warning – No immediate problem.
Low Minor problem – Possible service
impact.
Medium Minor problem – Minor service
impact.
High Major problem – Major service
impact.
Critical Emergency – System outage.
Id Integer 0 to A sequentially generated number
2147483647 that can be used to uniquely identify
the alarm.
<relation> Choice <= Less than or equal to
== Equal to
>= Greater than or equal to
HAS Contains

©
Variable Field Attribute Valid Values Description
<back> numAlarms Integer 1 to 10000 The number of alarms to move
back.
Type
Get
Result (sample provided)

Wed Mar 31 12:30:20 EDT 2010
idnum = 2375
trapName = bwSystemHealthReport
severity = Informational
trapType = Notification
component = Applicationserver
subcomponent = Processmonitor
systemHostName = viras01.mtl.broadsoft.com
problemText:
No abnormal condition detected.
recommendedActions:
None.
Type
get TrapSeverity <= Medium
Result (sample provided)

Thu Apr 01 07:29:03 EDT 2010
idnum = 2667
trapName = bwSipTcpSocketError
severity = Low
trapType = Notification
component = Applicationserver
subcomponent = Sip
systemHostName = viras01.mtl.broadsoft.com
problemText:
SIP TCP connection associated with 192.168.9.142:61620 was released
or unable to be established.
recommendedActions:
1 alarm(s) found
6.1.3 View and Set CLI Alarm Backlog Size

The SNMP agent backlog size determines how many alarms are stored on disk and thus
the maximum number of alarms that can be retrieved by CLI clients. When the actual
number of historical alarms overflows the size parameter, the oldest alarms are deleted.
Having a “localsize” more than “size” is allowed but it yields the same result as having
“localsize” equal to “size”. In other words, “localsize” is limited by “size”.
The Logging to File option is used to persist alarms across SNMP agent restarts.
The recommended backlog size is “1000” alarms.

©
6.1.3.1 View Current CLI Alarm Backlog Size
From the CLI Monitoring/Alarm level, type showconfig. The result displays (a sample is
provided).
Alarms-Related Configuration
Local backlog size (nb. alarms) = 1000
SNMP agent logging to file = enabled
SNMP agent backlog size = 1000
6.1.3.2 Set CLI Alarm Backlog Size

1) Make sure you are at the Monitoring/Alarm> level.
2) Enter:
set <attribute> ↵
… where:
Variable Field Type Valid Values Description
<attribute> localsize Integer 1 to 5000 Enter an integer, 1 to
5000.
The local alarms
backlog size (in number
of alarms).
size Integer 1 to 5000 Enter an integer, 1 to
50000.
The SNMP agent alarms
backlog size (in number
of alarms).
logging Choice disabled Logging to disk is off.
enabled Logging to disk is on.
6.1.4 Enable and Disable Real-Time Alarms Echoing

Use this command to enable or disable real-time alarm echoing. Alarms appear at the CLI
as they occur when the command is enabled.
6.1.4.1 Enable Real-time Alarms Echoing

From the CLI Monitoring/Alarm level, type show on. The result displays.
...Done
NOTE: When alarms occur, they appear in the window from which this command was invoked.
To avoid interrupting other CLI commands, open a separate window.
6.1.4.2 Disable Real-time Alarms Echoing

From the CLI Monitoring/Alarm level, type show off. The result displays (a sample is
provided).
...Done

©
6.1.5 Clear Alarms
Use this command to clear alarms in the CLI alarm buffer. From the CLI Monitoring/Alarm
level, type clear.
Example
clear
Result
...Done
6.1.6 Close System Alarms

Use this command to close the connection with the SNMP agent, to obtain system alarms
and event logs generated by BroadWorks.
6.1.6.1 Close Connection for Alarms

From the CLI Monitoring/Alarm level, type close. The result displays (sample provided).
...Done
6.2 Threshold Alarms

BroadWorks supports configurable threshold alarms. Thresholds provide a mechanism to
monitor the state of a BroadWorks server. With thresholds configured, a BroadWorks
server automatically sends an SNMP trap when an operator-specified limit is reached
against a specific BroadWorks counter or gauge. Thresholds can be assigned against
any counter, gauge, or column defined in a BroadWorks application MIB (applies to all
server types).
Thresholds are configurable through the CLI or via SNMP.
6.2.1 Threshold Attributes

There are two kinds of thresholds: counter thresholds and gauge thresholds.
The attributes of counter thresholds include:
 Supports many thresholds per counter.
 Supports activating and deactivating thresholds for a counter.
 A notification alarm is generated each time a counter crosses the current comparison
level of a threshold.
Each threshold level has the following attributes:
− Current comparison level – This level establishes when an alarm is generated.
It is not configurable by the service provider.
− Initial comparison level – This level provides the basis of comparison. Reset
the counter to 0 to compare to this level.
− Offset level – This level defines the increment added to the current comparison
value whenever the counter crosses the current comparison value. The offset
may be 0, meaning the threshold alarm is raised at most once, unless the counter
is reset.

©
− Severity level – This level defines the severity level included in each notification
alarm generated by this threshold.
NOTE: When the counter rolls over its maximum value, the current comparison level is also
rolled over, to continue generating the alarms at every “offset” increment.
The attributes of gauge thresholds include:

 Support for many thresholds per gauge.
 Support for activating and deactivating thresholds for a gauge.
 To avoid repeatedly generating alarms around a particular threshold level, each
threshold level has a NotifyHigh level and a NotifyLow level. The following attributes
are defined for each threshold:
− NotifyHigh comparison level – This gauge value establishes when an alarm is
generated for a gauge value in a rising direction. When subsequent similar
gauge values cross the NotifyHigh level, they do not generate an alarm until the
gauge has reached the corresponding NotifyLow level.
− NotifyLow comparison level – This gauge value establishes when an alarm is
generated for a gauge value in a falling direction. When subsequent similar
gauge values cross the NotifyLow level, they do not generate an alarm until the
gauge has reached the corresponding NotifyHigh level.
6.2.2 Configure Thresholds using CLI
6.2.2.1 Add New Counter Threshold

From the CLI Monitoring/Threshold level, type add counter “Sync API auth
failures” syncNbAuthorizationFailures 1 1 high active. The following
response appears (sample provided).
...Done
NS_CLI/Monitoring/Threshold> get counter

Description Name Initial Offset
Current
Severity Status
Sync API auth failures syncNbAuthorizationFailures 1 1
0
informational active
In this example, “Synch API auth failures” is the description that appears in the Problem
Text field of the threshold alarm and “syncNbAuthorizationFailures” is the counter
monitored by the threshold mechanism. The alarm raises with each new synch API
authorization failure (initial value = 1, offset value = 1).
6.2.2.2 Add New Gauge Threshold

From the CLI Monitoring/Threshold level, type add gauge “Unassigned numbers”
systemNbUnassignedDNs 100 200 informational active. The following
response appears (sample provided).
...Done
NS_CLI/Monitoring/Threshold> get gauge

©
Description Name Low High Severity
Status
Unassigned numbers systemNbUnassignedDNs 100 200 informational
active
In this example, “Unassigned numbers” is the description that appears in the Problem Text
field of the threshold alarm and “systemNbUnassignedDNs” is the value of the threshold
gauge. The alarm is raised whenever the number of unassigned numbers goes outside of
the range (100, 200).
6.2.2.3 Delete Threshold

From the CLI Monitoring/Threshold level, type:
delete counter syncNbAuthorizationFailures
or
delete gauge systemNbUnassignedDNs
The following response appears.

...Done
NOTE: If there are multiple thresholds for the same counter/gauge, then a list of thresholds
appear with the corresponding indexes. Use the Delete command (again) with the appropriate
index as an additional parameter on the command line.
6.2.2.4 Activate and Deactivate a Threshold

From the CLI Monitoring/Threshold level, type:
set counter syncNbAuthorizationFailures status inactive,
or
set gauge systemNbUnassignedDNs status inactive
The following response appears.

...Done
NOTE: If there are multiple thresholds for the same counter/gauge, then a list of thresholds
appear with the corresponding indexes. Use the Set command (again) with the appropriate
index as an additional parameter on the command line. For example, set counter
syncNbAuthorizationFailures status inactive rowIndex 0.

©
6.2.2.5 Invalid Thresholds
Thresholds that are in an active state contain valid parameters. Thresholds that are in an
invalid (4) state contain invalid parameters. Check the status of a threshold. You set it to
the active state to validate it.
Possible causes for a threshold being invalid include:
 The counter or gauge specified does not exist.
 The offset value of a counter threshold is not strictly positive (>0).
 The NotifyLow of a gauge threshold is higher or equal than its NotifyHigh.
6.2.3 Configure Thresholds through SNMP

The threshold SNMP MIB is described in the BroadWorksMaintenance.mib file distributed
with BroadWorks.
The following threshold tables are located in enterprises.broadsoft.broadworks.
common.thresholds:
 thCounterModule.bwCounterThresholdTable
 thGaugeModule.bwGaugeThresholdTable
The following sections explain how to create, delete, and control the thresholds. For more
information on the meaning of each column in the counter or gauge threshold tables, see
the BroadSoft Managed Objects Functional Specification (see your BroadSoft
representative).
6.2.3.1 Create New Threshold

To create a new threshold, create a new row in the threshold table. Set the Control
column of the row to “create (3)”. The Control column is
bwCounterThresholdControl for the counter threshold table and
bwGaugeThresholdControl for the gauge threshold table. The row is specified using
the Index column (bwCounterThresholdIndex or bwGaugeThresholdIndex,
respectively). The index is valid. The Control column takes the “inactive (0)” value. The
threshold is inactive by default. You can set other columns before you activate the
threshold.
6.2.3.2 Delete Threshold

To delete a threshold, type a “delete (2)” value in the Control column. Once a row is
deleted, it is not accessible.
6.2.3.3 Activate or Deactivate Threshold

To activate a threshold, type the “active (1)” value in the Control column. To deactivate a
threshold without deleting it, type the “inactive (0)” value in the Control column.
6.2.3.4 Column Descriptions

Description Column – Whatever is in the description column is appended to the alarms
generated by the threshold. Use the description column to assess why an alarm was
generated. Example descriptions include:
 Abnormally high number of MGCP restarts
 Database checkpoints failed too many times

©
Name Column – Use the name column to identify the counter or gauge to be monitored
by the threshold.
NOTE: This is not the name of the threshold.
For instance, the following names align with the examples provided in the example
included in the previous description column:
 MGCPStatsRestartInProgressIns
 ttNbFailedCheckpoints
6.2.4 Application Server Thresholds Limitation

The Application Server is internally divided in two processes: the Execution Server (XS)
and the Provisioning Server (PS). Some counters and gauges are located in the XS,
whereas others are located in the PS. Under normal circumstances, the Application
Server determines to which process the counter or gauge is connected. It ensures the
right process generates the alarm when the threshold is reached.
A few counters and gauges are defined in both the Execution Server and Provisioning
Server. These counters and gauges may have different names in the Management
Information Base (MIB), but their internal names are the same in both processes. The
threshold sub-system cannot determine on which server they are located.
This has two results:
 A valid Provisioning Server counter or gauge does not cause the threshold to become
“invalid” when activated and therefore, does not generate threshold alarms.
 A valid Execution Server counter or gauge generates threshold alarms for both the
Provisioning Server and Execution Server without discrimination. For example, the
Execution Server generates a threshold alarm and the current value of the threshold
is updated. Subsequently, the Provisioning Server generates a threshold alarm
based on the new value. The current value continues to be updated without
intervention.
An example of a counter is ttNbBackdoorUpdates or psTtNbBackdoorUpdates. In this
case:
 If you create a threshold on psTtNbBackdoorUpdates, it does not have an effect.
The threshold can be “active”.
 If you create a threshold on ttNbBackdoorUpdates, it has the effect described above.
The name of the Provisioning Server version differs from the Execution Server version
only by the prefix.

©
6.3 Performance Measurements
Performance measurements are organized in a tree structure. Commands are provided
to navigate, view the next, view the current position of, and view the values for MIB nodes.
NOTE: The local host address (127.0.0.1) must be included in the SNMP Agent’s access list.
6.3.1 Navigate through MIB Nodes

Use this command to navigate through the MIB nodes in a manner similar to the standard
cd (change directory) command. Use the slash (/) parameter to navigate back to root.
Use the (..) parameter to go back one level.
6.3.1.1 Navigate MIB Nodes

From the CLI Monitoring/PM/Execution or from Monitoring/PM/Provisioning (for the
Application Server) level, type cd [<path>]. Valid values include:
<path> String 0 to 1023 characters The path of the desired
node.
6.3.1.2 Examples
Type:
cd /system
Response (sample provided)

executionServer/systemModule
6.3.2 View Next MIB Node

Use this command to view the next MIB node in a manner similar to the standard ls (list)
command.
Application Server) level, type ls [<option>] [<path>]. Valid values include:
<option> Choice -r Shows the tree structure of the
MIB nodes.
-n Shows the name of the nodes
containing values.
<path> String 0 to 1023 characters Shows the path to the node you
want to view.

©
6.3.2.1 Examples
Type:
AS_CLI/Monitoring/PM/Execution> ls –r

mgcpModule
mgcpStats
imsModule
imsStats
databaseModule
databaseStats
timesTen
xsRemoteXla
callpModule
callpStats
smtpModule
smtpStats
sipModule
sipStats
congestionManagement
services
accountCodes
anonymousCallRejection
6.3.3 View Current Position in MIB

Use this command to view the current position in the MIB in a manner similar to the
standard pwd (present working directory) command.
Application Server) level, type:
pwd

executionServer
6.3.4 View MIB Node Values

Use this command to view MIB node values. MIB node values can be viewed in the
following ways:
 All the performance measurements throughout the MIB nodes tree structure
 The performance measurements related to the current node
 The performance measurements of a specified path
From the CLI Monitoring/PM level, type get [<option>] [<path>] [<tableKey>].

©
Valid values include:
<option> Choice -r Displays all performance
measurements for the current tree
structure of the MIB nodes.
-f Displays a full description of
performance measurements.
<path> String 0 to 1023 Displays the path of the node for
characters which you want to view performance
measurements.
<tableKey> String 0 to 127 characters Used to specify a row key when the
node is displayed as a table.
6.3.4.1 Examples
Type:
Get
-------------------------------------------------------------------------
executionServer/callpModule/callpStats/
-------------------------------------------------------------------------
*bwCallpNetworkOriginationAttempts 4
*bwCallpNetworkTerminationAttempts 5
*bwCallpNetworkTerminationsAnswered 4
*bwCallpUserOriginationAttempts 6
*bwCallpUserTerminationAttempts 9
*bwCallpUserTerminationsAnswered 7
*bwCallpEmergencyCallAttempts 0
*bwCallpEmergencyCallAlarms 0

©
7 Server Management
This chapter describes server management for the BroadWorks servers.
7.1 Server State Management
7.1.1 Server State

BroadWorks servers support an administrative state.
The server administrative states include:
 Locked – No traffic is allowed through the server.
 Unlocked – Traffic is allowed through the server.
The server is operating normally when it is in the Unlocked administrative state.
BroadWorks servers are in a Locked state while they are rebooting and they transition to
the Unlocked state when they are fully initialized.
7.1.2 Application State

Each BroadWorks application implements its own state. The application state is
presented as an administrative state and an effective state. Both states can be viewed
from the CLI Maintenance/ManagedObject level using the “get broadworks full” command.
Only the administrative state can be controlled by an operator; the effective state simply
reflects the actual state of the application. The administrative state is controlled using the
lock and unlock commands at the CLI Maintenance/ManagedObject level. As the
commands suggest, the possible administrative states are:
 Locked
 Unlocked
When an application administrative state is set to one of these states, it means that the
application tries to transition to that state (assuming the application is running).
On the other hand, the effective state reflects the actual state of the application and can
have the following values.
 Locked: The application is locked. Depending on the application’s implementation, it
may mean that it provides limited to no functionality at all.
 Locking: The application is in the process of being locked. This is a transitional state.
 Unlocked: The application is unlocked; all its functionality is available.
 Unlocking: The application is in the process of being unlocked. This is a transitional
state.
 Stopping: The application is in the process of being stopped. This is a transitional
state.
 Stopped: The application is stopped; none of its functionality is available.
 InTrouble: The application is not functional; one or more (but not all) of its containers
is dead. The application must be restarted. The application also enters this state
when one or more (but not all) of its containers is restarted by the process monitor.
 Dead: The application is not functional; all of its containers are dead. The application
must be restarted.

©
Following is an example of the get broadworks full command, where four
BroadWorks applications are deployed but only two are running.
AS_CLI/Maintenance/ManagedObjects> get broadworks full
BroadWorks Managed Objects
==========================
* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
Administrative State..: Unlocked
* Applications:
Name Version Deployed Administrative State Effective
State
===================================================================================
====
ExecutionAndProvisioning 17.0_1.446 true Unlocked
Unlocked
FlashPolicy 17.0_1.446 true Unlocked
Stopped
OpenClientServer 17.0_1.446 true Unlocked
Stopped
WebContainer 17.0_1.446 true Unlocked
Unlocked
4 entries found.
* Hosted Applications:
Name Version Context Path Deployed
================================================================
CommPilot 17.0_1.446 /CommPilot true
DeviceManagementFiles 17.0_1.446 /DeviceManagement true
JWSFiles 17.0_1.446 /FileRepos true
MediaFiles 17.0_1.446 /media true
OCIFiles 17.0_1.446 /ocifiles true
5 entries found.

apache (pid=25253)
tomcat (pid=25189)


©
The server administrative state, as shown at the top of the above output as Unlocked, also
has an impact on the application’s effective state. For instance, if an operator locks the
server, all application effective states transition to Locked regardless of the application’s
administrative state. This is because the Locked server’s administrative state supersedes
the application’s administrative state. However, when the server is in Unlocked state, an
operator can freely change an application administrative state and the application effective
state transitions to the requested state. Following is an example where the server
administrative state is set to “Locked” and the application’s administrative state is
“Unlocked” but the effective state of the running application is “Locked”.
==========================
* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
Administrative State..: Locked
* Applications:
Name Version Deployed Administrative State Effective
State
===================================================================================
====
Locked
FlashPolicy 17.0_1.446 false Unlocked
Stopped
OpenClientServer 17.0_1.446 false Unlocked
Stopped
Locked
4 entries found.
================================================================
5 entries found.

apache (pid=25253)
tomcat (pid=25189)


©
7.1.3 Get Server and Application States
NOTE: The CLI can be used to query the server and application states without the server
running and without logging in to the CLI.
From the CLI Maintenance/ManagedObjects level, type get broadworks. The result
displays (sample provided).
==========================
* Server:
Identity..............: AS
Version...............: Rel_17.0_1.430
* Applications:
Name Version Deployed Administrative State Effective State
=======================================================================================
ExecutionAndProvisioning 17.0_1.430 true Unlocked Unlocked
FlashPolicy 17.0_1.430 true Unlocked Unlocked
OpenClientServer 17.0_1.430 true Unlocked Unlocked
WebContainer 17.0_1.430 true Unlocked Unlocked
4 entries found.
================================================================
5 entries found.
7.2 Application Management

Operators can install (on the Xtended Services Platform only), activate, and deploy
applications on BroadWorks servers. The servers provide Application Management
functionality through the CLI, under the Maintenance/ManagedObjects level. Applications
can be installed, uninstalled, activated, deactivated, deployed, and undeployed. All
operations are CLI-driven. No manual operation is required.
Applications must be packaged using two possible formats, the standard web application
archive (.war) format, and the BroadWorks Application format (.bwar). The web
application must follow specific guidelines documented in the BroadWorks Xtended
Services Platform Configuration Guide so that they can be installed on the Xtended
Services Platform. The BroadWorks applications are created and released by BroadSoft.
The full life cycle of an application is:
Installation  Activation  Deployment  Undeployment  Deactivation  Uninstallation
Individual CLI operations are described in the following subsections. Unless otherwise
mentioned, no restart is necessary for any operation. All changes are effective
immediately.

©
7.2.1 Installation
NOTE: Installation of applications can only be performed on the Xtended Services Platform.
The first step in the application life cycle is the installation. For the installation of
applications that are not prepackaged with the Xtended Services Platform, the application
file (.war or .bwar) must be copied to a local temporary directory on the server, for
example, /tmp. Using the CLI, in the Maintenance/ManagedObjects level, invoke the
install command, providing the full path name of the application file. Note that the name of
the file is irrelevant to the Xtended Services Platform. All the descriptive and operational
data used by the Xtended Services Platform are inside the file. By default, prepackaged
applications are located in the /usr/local/broadworks/apps directory.
An application is made unique by its name and its version, both of which are defined in the
internal deployment descriptor (in the .war/.bwar). For web application, the deployment
descriptor is the web.xml file. On the other hand, the BroadWorks application deployment
descriptor is in the broadworks.xml file. The Xtended Services Platform enforces
uniqueness of applications and does not allow installation of two applications with the
same name/version.
Following is an example of an installation of an application.
XSP_CLI/Maintenance/ManagedObjects> install application /tmp/Xsi-
Actions_5.0.war
Broadworks SW Manager installing Xsi-Actions_5.0.war...
- Performing basic validation
. Valid application structure
. Name : Xsi-Actions
. Description: BroadWorks Xtended Services Interface
. Version : 5.0
...Done
XSP_CLI/Maintenance/ManagedObjects>
During the installation process, the application file is validated and, if valid, it is internally
copied and installed. Once the application is installed, the file used to install the
application is no longer required and can be deleted.
After being installed, either the application can be activated to make it configurable or it
can be uninstalled.
NOTE: Applications with an automatic upgrade mode are installed by default when the
BroadWorks server is installed. Such applications cannot be uninstalled.

©
7.2.2 Activation
To be able to configure the application, it must be activated. Activation is performed
through the CLI with the activate command. The following examples illustrate the
activation of applications.
AS_CLI/Maintenance/ManagedObjects> activate ?
<type>, Choice = {application}
application:
<name>, String {1 to 80 characters}
<version>, String {1 to 80 characters}
[<contextPath>, String {1 to 80 characters}]
AS_CLI/Maintenance/ManagedObjects> activate application OpenClientServer

17.0_1.446
BroadWorks SW Manager activating...OpenClientServer version 17.0_1.446
...Done
AS_CLI/Maintenance/ManagedObjects> activate application OCIOverSoap

17.0_1.446 /ociSOAP
BroadWorks SW Manager activating...OCIOverSoap version 17.0_1.446
...Done
When activating an application, the name and version are mandatory. The contextPath
parameter is only required when activating a web application. It must not be specified
when activating a BroadWorks application.
Multiple instances of the same web application can be activated using different context
paths. However, a single activation of a BroadWorks application is allowed.
Once activated, applications can be configured. Configuration is done through the CLI
from the application-specific context. This context can be found under the application
context as illustrated in the following example.
AS_CLI/Applications> ?
0) CommPilot : go to level CommPilot
1) DeviceManagementFiles : go to level DeviceManagementFiles
2) ExecutionAndProvisioning : go to level ExecutionAndProvisioning
3) JWSFiles : go to level JWSFiles
4) MediaFiles : go to level MediaFiles
5) OCIFiles : go to level OCIFiles
6) OCIOverSoap : go to level OCIOverSoap
7) OpenClientServer : go to level OpenClientServer

(repeat)
AS_CLI/Applications/CommPilot> ?
0) CCRSProxyUrls : go to level CCRSProxyUrls
1) GeneralSettings : go to level GeneralSettings
2) Logging : go to level Logging
3) RelativeUrl : go to level RelativeUrl
4) WebBranding : go to level WebBranding

(repeat)
As shown in the example, all the activated applications are listed as subcontext. Note
however, that not all applications support configuration.

©
7.2.3 Deployment
Applications activated on the server can be selectively deployed using the CLI deploy
command. Following are examples of application deployments.
XSP_CLI/Maintenance/ManagedObjects> deploy ?
<type>, Choice = {application}
application:
<nameOrContextPath>, String {1 to 255 characters}
AS_CLI/Maintenance/ManagedObjects> deploy application /ociSOAP

BroadWorks SW Manager deploying /ociSOAP...
...Done
AS_CLI/Maintenance/ManagedObjects> deploy application FlashPolicy

BroadWorks SW Manager deploying FlashPolicy...
...Done
AS_CLI/Maintenance/ManagedObjects>
As shown in the previous example, deploying a web application and a BroadWorks

application requires different parameters. A web application is deployed using the context
path specified during the activation. On the other hand, deploying a BroadWorks
application is done using the application name.
The following subsections present the deployment differences between web and
BroadWorks application.
7.2.4 Undeployment
Applications can be undeployed when they are no longer required. Web applications can
be undeployed without restriction; however, before being undeployed, BroadWorks
applications must be stopped. Undeploying an application is done with the command
undeploy invoked from the CLI level Maintenance/ManagedObjects. For web
applications, the context path must be specified and for BroadWorks applications, the
application name must be used. Following are some examples.
AS_CLI/Maintenance/ManagedObjects> undeploy application FlashPolicy
BroadWorks SW Manager un-deploying FlashPolicy...
...Done
AS_CLI/Maintenance/ManagedObjects> undeploy application /ociSOAP

BroadWorks SW Manager un-deploying /ociSOAP...
...Done

==========================
* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
* Applications:
Name Version Deployed Administrative State
Effective State
=========================================================================
==============
Unlocked

©
FlashPolicy 17.0_1.446 false Unlocked
Stopped
OpenClientServer 17.0_1.446 false Unlocked
Stopped
Unlocked
4 entries found.
================================================================
OCIOverSoap 17.0_1.446 /ociSOAP false
6 entries found.

apache (pid=25253)
tomcat (pid=25189)

Once a web application is undeployed, no instance of this application’s context path is

running within the WebContainer. To reuse the context path, the web application must be
deactivated.
When a BroadWorks application is undeployed, it can no longer be started.
Applications can still be configured once undeployed.
NOTE: Undeploying the BroadWorks Web Container application affects all web applications.
The web applications are no longer running.

©
7.2.5 Deactivation
Undeployed applications can be deactivated using the deactivate CLI command.
Deactivated applications can no longer be configured. As such, their associated CLI level
under the application context is removed. For web applications, the context path is
released and can be reused. Following are some examples of application deactivation.
Note that deactivating a BroadWorks application is done using the application name,
whereas deactivating a web application is done using the context path.
AS_CLI/Maintenance/ManagedObjects> deactivate application
OpenClientServer
BroadWorks SW Manager deactivating...OpenClientServer version 17.0_1.446
...Done
AS_CLI/Maintenance/ManagedObjects> deactivate application /ociSOAP

BroadWorks SW Manager deactivating...OCIOverSoap version 17.0_1.446
...Done
7.2.6 Uninstallation
NOTE: Only the Xtended Services Platform supports uninstallation of applications.
Installed applications that are no longer required can be uninstalled. This removes all files
related to this application from the server. Only applications in Installed state can be
uninstalled. Furthermore, only applications having their upgrade mode set to “Manual”
can be uninstalled. Those with an automatic upgrade mode cannot be uninstalled as they
are part of the server configuration and they are upgraded with the platform. An
application is uninstalled using the uninstall command form the CLI, in the
Maintenance/ManagedObjects level. The uninstall command requires an application and
a version. Following is an example of an uninstallation of a web application.
XSP_CLI/Maintenance/ManagedObjects> uninstall application Xsi-Actions 5.0
Broadworks SW Manager un-installing Xsi-Actions version 5.0...
...Done
XSP_CLI/Maintenance/ManagedObjects>
7.3 Start Server

The CLI can be used to start a server.
NOTE 1: It is not necessary to log in to the CLI. The server automatically goes to the Unlocked
administrative state after starting.
NOTE 2: The server can also be started from the UNIX prompt as bwadmin using the startbw
command.
On the Application Server and Network Server, a check of the database replication lagging
state for the cluster occurs at startup. For a description of the database replication lagging
state, see section 11.1.1 TimesTen Database Synchronization.

©
If a database replication lagging condition exists for any peer in the cluster, a warning is
printed.
7.3.1 Start BroadWorks

From the CLI Maintenance/ManagedObjects level, type start. The following results
appear.
AS_CLI/Maintenance/ManagedObjects> start
--> Starting BroadWorks <--
Broadcast message from bworks (Thu Apr 1 15:35:34 2010):
===== BROADWORKS CONTROL --- START INITIATED =====

Starting [done]
7.3.2 Start BroadWorks Application

It is possible to start a single application using the start application
<application> command.
AS_CLI/Maintenance/ManagedObjects> start application FlashPolicy
--> Starting application FlashPolicy <--
Broadcast message from bworks (Tue Apr 6 07:46:27 2010):
===== BROADWORKS CONTROL --- START INITIATED (FlashPolicy) =====

Starting [done]
7.4 Stop Server
WARNING: Stopping a server results in that server refusing any new incoming requests. This
puts the server in an administrative Disabled state.
The BroadWorks server can be stopped at either the UNIX prompt or CLI.
NOTE: It is not necessary to log in to the CLI. The server cannot be stopped from the CLI
unless a traffic lock has first been performed.
7.4.1 Stop BroadWorks

1) From the CLI Maintenance/ManagedObjects level, type lock. The server
transitions from the Unlocked to Locking state before finally entering the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
that the server administrative state is Locked.

©
NOTE: A server transitions to the locked state once all active sessions are terminated. Use the
lock force command form the CLI Maintenance/ManagedObjects level to force the
administration state to Locked.
3) From the CLI Maintenance/ManagedObjects level, type stop. The results display
(sample provided).
AS_CLI/Maintenance/ManagedObjects> stop
+++ WARNING +++ WARNING +++ WARNING +++

Stopping the server and/or application will cause downtime for the
targeted component.
Continue?
Please confirm (Yes, Y, No, N): y

--> Stopping BroadWorks <--
===== BROADWORKS CONTROL --- STOP INITIATED =====

Stopping [done]
7.4.2 Stop Server without Locking

Specify the force flag to terminate all active sessions.
From the CLI Maintenance/ManagedObjects level, type stop force. The results
display (sample provided).
AS_CLI/Maintenance/ManagedObjects> stop force
--> Stopping BroadWorks <--

Stopping [done]
7.4.3 Stop BroadWorks Application

A single application can be stopped (with and without locking) from the CLI using the
command stop [force] application <application>.
AS_CLI/Maintenance/ManagedObjects> stop force application FlashPolicy
--> Stopping application FlashPolicy <--
Broadcast message from bworks (Tue Apr 6 07:50:26 2010):
===== BROADWORKS CONTROL --- STOP INITIATED (FlashPolicy) =====

Stopping [done]

©
7.5 Check for Running Processes
Use this command to see all BroadWorks-related running processes.
From the CLI Maintenance/ManagedObjects level, type get broadworks full. The
following results appear (sample provided).
==========================
* Server:
Identity..............: AS
Version...............: Rel_17.0_1.440
* Applications:
Name Version Deployed Administrative State Effective State
=======================================================================================
ExecutionAndProvisioning 17.0_1.440 true Unlocked Unlocked
FlashPolicy 17.0_1.440 false Unlocked Stopped
OpenClientServer 17.0_1.440 false Unlocked Stopped
WebContainer 17.0_1.440 true Unlocked Unlocked
4 entries found.
================================================================
5 entries found.

apache (pid=3234)
tomcat (pid=3174)

NOTE: From the UNIX prompt, type the showrun command to view BroadWorks-related
processes.

©
7.6 Lock Network Server, Application Server, or Execution Server
Use this command to put the server in the traffic Locked state to make sure that new
sessions are not accepted, and the server waits until all active sessions are cleared before
a server restarts or stops.
WARNING: Locking a server results in that server refusing any new incoming requests. This
takes the server operationally out-of-service.
NOTE: Configuration changes can be made while the server is in the Locked state (unless the
database is locked using a separate procedure, if applicable).
From the CLI Maintenance/ManagedObjects level, type lock. The following response
appears (sample provided).
This command will lock the server. Note that this action could cause
downtime. Continue?
Please confirm (Yes, Y, No, N): y

…Done
The server is in the Locking administrative state while there are sessions in progress. No
new sessions can be started though. The server becomes Locked when the last session
terminates.
The server administrative state can be verified at the Maintenance/ManagedObjects level
by typing get broadworks.
7.6.1 Force Lock Server Immediately

From the CLI Maintenance/ManagedObjects level, type lock force. The results
display (sample provided).
…Done
The server administrative state can be verified using the get broadworks command.
7.6.2 Unlock Server

Use this command to unlock the server.
From the CLI Maintenance/ManagedObjects level, type unlock. The results display
(sample provided).
…Done

©
7.7 Restart Server
It is recommended to restart a server only when in the Locked administrative state. If an
immediate restart is necessary, the “force” flag can be used. The server automatically
goes to the Unlocked administrative state after restarting.
WARNING: Restarting a server results in that server refusing any new incoming requests until
the server restarts.
7.8 Take BroadWorks Server Out of Service for Maintenance

At times, it is necessary to take a BroadWorks server out of service to perform server
maintenance or to power down the server.
7.8.1 Non-redundant Systems

Use this procedure to take a non-redundant server out of service:
1) From the CLI Maintenance/ManagedObjects level, type lock. The server
transitions from the Unlocked to Locking state before it enters the Locked state.
the server is in the Locked state.
NOTE: A server transitions to the Locked state only once all active sessions are terminated.
Use the lock force command from the CLI Maintenance/ManagedObjects level to force the
administration state to go to Locked.
3) From the CLI Maintenance/ManagedObjects level, type stop. The server is now
prepared to power down. If a power down is required, the following step should be
followed to ensure a graceful shutdown of the server.
4) Log in (or su -) as root.
5) At the UNIX prompt, type # sync.
6) At the UNIX prompt, type # shutdown –y –g0 –i5. The server powers down.
7.8.2 Redundant Systems

Use the following procedure to take a redundant server out-of-service. The procedure
applies to any cluster peer (for example, primary or secondary Application Server). The
remaining peer handles all traffic.
1) From the CLI Maintenance/ManagedObjects level, type lock. This ensures the
remaining in-service cluster peer handles all new calls. The server transitions from
the Unlocked to Locking state before it enters the Locked state.
the server is in the administration Locked state.
NOTE: Terminate all active sessions to allow the server to make the transition to the Locked
state. Use the lock force command from the CLI Maintenance/ManagedObjects level to force
the administration state to Locked.

©
3) From the CLI System/Peering level, type lock. This ensures changes are not made
to the database for the server under maintenance. The database transitions from the
Unlocked to Locking state before it enters the Locked state.
NOTE: This step is for Application Server and Network Server only.
4) From the CLI System/Peering/Peers level, type get to verify the server database
state is Locked.
5) the CLI Maintenance/ManagedObjects level, type stop.

6) From the UNIX prompt, type $ repctl stop.
7) The server is now prepared to power down. If a power down is required, the following
step should be followed to ensure a graceful shutdown of the server.
8) Log in (or su -) as root.
9) At the UNIX prompt, type # sync.
10) At the UNIX prompt, type # shutdown –y –g0 –i5.
7.9 Restore BroadWorks Server to In-service

It is important to properly return servers to in-service.
7.9.1 Non-redundant Systems

Use the following procedure to restore a non-redundant server back to in-service:
1) Power up the server. The BroadWorks application starts automatically.
2) Log in as bwadmin.
3) From the CLI Maintenance/ManagedObjects level, type get broadworks full.
4) Verify that the administration state is Unlocked.
NOTE: If the server was locked prior to being shut down, its state is still Locked after the power-
up as the administration state is persisted. Unlock the server using the unlock command.
5) From the CLI Maintenance/Tools level, type healthmon. Report severity displays
“Notification”, indicating that all BroadWorks subcomponents are up and running
properly.

©
7.9.2 Redundant Systems
Use the following procedure to return a member of a redundant server cluster to
in-service. This procedure applies to any cluster peer (including the primary or secondary
Application Server).
1) Power up the server. The BroadWorks application starts up automatically.
2) Log in as bwadmin.
3) From the CLI Maintenance/ManagedObjects level, type get broadworks full.
4) Verify the administration state is Unlocked.
NOTE: If the server was locked prior to being shut down, its state is still Locked after the power-
up as the administration state is persisted. Unlock the server using the unlock command.
5) From the CLI System/Peering/Peers level, type get to verify the server database
state is Locked.
6) From the CLI Maintenance/Tools level, type healthmon. Assuming normal server
conditions, the report severity displays “Notification”, indicating that all BroadWorks
subcomponents are up and running properly.
7) From the CLI Maintenance/ManagedObjects level, type lock to make sure that new
calls are handled by the remaining in service cluster peer. The server transitions from
the Unlocked to Locking state before finally entering the Locked state.
that the server administration state is Locked.
state. Use the lock force command from the CLI Maintenance/ManagedObjects level to
force the administration state to Locked.
9) From the CLI Maintenance/ManagedObjects level, type stop.

10) From the UNIX prompt, type $ repctl stop to stop replication.
11) From the CLI Maintenance/Tools level, type importdb <remote server
hostaddress> to import the database from the peer server.
NOTE: Because the BroadWorks application has been stopped, the CLI login command fails.
The Maintenance/Tools CLI level is accessible without CLI login. The importdb <remote server
hostaddress> parameter must be the same as the UNIX host name of the active server.
12) From the UNIX prompt, to start replication, type $ repctl start.
13) From the UNIX prompt, to unlock the database, type $ peerctl unlock.

©
14) From the CLI Maintenance/ManagedObjects level, type start to start the server.
NOTE: Because the BroadWorks application is stopped, the CLI login command fails. The
Maintenance/ManagedObjects CLI level is accessible without CLI login. The server can also be
restarted from the UNIX prompt with the startbw command.
15) From the CLI Maintenance/ManagedObjects level, type unlock to make sure the
server can accept new calls. The server transitions from the Locked to Unlocking
state before finally entering the Unlocked state.
16) From the CLI Maintenance/ManagedObjects level, type get broadworks full
Verify the administration state is Unlocked and the operational state is Enabled.
17) From the CLI System/Peering/Peers level, type get to verify that the server database
state is Unlocked.
18) From the CLI Maintenance/Tools level, type healthmon to verify the report severity
displays “Notification”, indicating that all BroadWorks subcomponents are up and
running properly.
7.10 Server State Monitoring Tools
7.10.1 Healthmon
The healthmon tool is available on all BroadWorks servers. You use this tool to report the
list of processes in trouble, if any, and monitor key system components. For the
Application Server and Network Server, healthmon also monitors the state of the database
and redundancy. Database out-of-sync condition detection, database replication lagging
detection, and database notification lagging detection are part of the healthmon command.
NOTE: The tool can be used in two distinct modes: Report to console or report over SNMP. In
the latter case, an SNMP trap is sent to the list of managers configured on the server. The
severity of the trap varies based on the state of the system. A trap is sent when no failure
conditions are detected. (Trap severity is NOTIFICATION.) This mechanism can be invoked
through a Solaris cron job.
MTLAS01$ healthmon –h
Usage : healthmon <-h|-v|-s|-l> [-p] [-d]

Description : This tool is used to monitor and report the status of a
BroadWorks
server. The report can either be generated locally at
the console
or sent over the network through an SNMP TRAP.

©
Note that a TRAP is still sent if no problems are
detected (keepalive).
Options
-h : print this usage note
-v : print the version of the script
-s : Generate a report and send the result through an SNMP
trap.
-l : Generate a local to screen.
-p : Test only the disk partitions status.
-d : Generate a detailed trace of all the tests executed.
This option is ignored in case of an SNMP report.
Some of the healthmon parameters can be manually configured by editing the

/bw/local/broadworks/bw_base/conf/healthmon.properties file. The settings stored in this
file are preserved during an upgrade. The following table describes the supported
parameters in healthmon.properties.
Parameter Description Default Value
WarningPartitionUsage If the occupancy on any disk partition in percentage reaches 80

this threshold value, a low severity trap is generated and the
action specified by parameter WarningPartitionActions is
performed.
WarningPartition Action performed if the occupancy on any disk partition run_autocleanup,

Actions reaches the value specified by parameter minimal_logging
WarningPartitionUsage.
HighPartitionUsage If the occupancy on any disk partition in percentage reaches 90

this threshold value, a high severity trap is generated and
the action specified by parameter HighPartitionActions is
performed.
HighPartitionActions Action performed if the occupancy on any disk partition run_autocleanup,

reaches the value specified by the parameter no_logging
HighPartitionUsage.
CriticalPartitionUsage If the occupancy on any disk partition in percentage reaches 95

this threshold value, a critical severity trap is generated and
the action specified by parameter CriticalPartitionActions is
performed.
CriticalPartitionActions Action performed if the occupancy on any disk partition run_autocleanup,

reaches the value specified by parameter no_logging
CriticalPartitionUsage.
StopBwPartionUsage If the occupancy on any disk partition in percentage (except 98

/persistentLogs partition) reaches this threshold value, a
critical severity trap is generated and the action specified by
parameter StopBwPartitionActions is performed.
StopBwPartionActions Action performed if the occupancy on any disk partition run_autocleanup,

(except /persistentLogs partition) reaches the value stopbw
specified by parameter StopBwPartitionUsage.
The default value is stopping BroadWorks application to
avoid possible database corruption.
WarningPermUsage If the TimesTen permanent memory area in use reaches this 80

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.
HighPermUsage If the TimesTen permanent memory area in use reaches this 90


©
Parameter Description Default Value
CriticalPermUsage If the TimesTen permanent memory area in use reaches this 95

WarningTempUsage If the TimesTen temporary memory area in use reaches this 80

HighTempUsage If the TimesTen temporary memory area in use reaches this 90

CriticalTempUsage If the TimesTen temporary memory area in use reaches this 95

StopBwPartionUsage./ If the occupancy on /persistentLogs partition in percentage 98

persistentLogs reaches this threshold value, a critical severity trap is
generated and the action specified by the parameter
StopBwPartionActions./persistentLogs performed.
StopBwPartionActions./ Action performed if the occupancy on /persistentLogs stop_replication

persistentLogs partition reaches the value specified by the parameter
StopBwPartionUsage./persistentLogs.
The default value is stopping replication.
CallLogsMaxUsageOn Threshold value in percentage. If the directory for offline 5

Partition CDR file queuing (/var/broadworks/billing/rf/) takes more
than this value of the available space on the partition where
it is mounted, file queuing is disabled.
Depending on the server type, different actions could be set for the Actions parameters in
healthmon.properties. The following table lists the supported actions.
Action Supported Servers
minimal_logging Application Server, Network Server, Execution Server
no_logging Application Server, Network Server, Execution Server
no_billing Application Server
run_autocleanup All servers
stopbw All servers
stop_replication All servers
7.10.2 Tech-support
The tech-support tool is available on all BroadWorks servers. You use this tool to report
the server software version, BroadWorks and Solaris processes status, Internet Protocol
(IP) configuration, redundancy status, and other diagnostics.
Log in as bwadmin to access this tool. The –help option provides the command format
information.
Usage: tech-support [-help|-day <day of month>|-dailyDump]

<no option> : data dump with today's sar and gc output - recommended
-help : print this usage note
-day <day of month> : data dump with specified day's sar and gc output
-dailyDump : performs a tech-support and puts the output in
/var/broadworks/logs/monitoring

©
-perf : dump performance information only
7.11 Server Maintenance State

During the application of some maintenance tasks, the server goes into an internal
maintenance state. This state is used to prevent major concurrent maintenance tasks
from being run. For example, it prevents an operator from accidentally starting a server
while a new software version is being activated. The following maintenance tasks cause
the server to go into a maintenance state:
 New software version activation (upgrade)
 Patching
 Database import
 Server reset, start, or stop operations
7.11.1 View Current Maintenance State

An operator can view whether or not a server is in maintenance state by running the
following command.
MoExtensions.pl –getMaintState
MTLAS01$ MoExtensions.pl –getMaintState

– RUNNING
7.11.2 Force Server into Maintenance State

An operator can manually force a server to go into a maintenance state by using the
following command as bwadmin. Note that this does not stop the application if it is
currently running.
MoExtensions.pl –setMaintState 9
7.11.3 Recover Server in Maintenance State

A server could remain in the maintenance state due to the improper termination of a
maintenance task. This causes most maintenance tasks to fail, for example, healthmon.
If there is no ongoing maintenance and the server has remained in the maintenance state,
the state can be changed by executing the following command.
MoExtensions.pl –setMaintState 7
This moves the server back into a running state.

©
8 Software Version Management
This section provides procedures to manage software versions. In this section all
examples provided are for the Network Server; however they also apply to BroadWorks
servers.
NOTE: The procedure to upgrade the server with a new software version is described in the
Installation and Upgrade Guide for that particular server. The final step of an upgrade procedure
can be performed from the Maintenance/ManagedObjects CLI level. Then the new version can
be activated or a previous version can be reactivated should there be a problem with the new
version.
8.1 Display Active Software Version
8.1.1 Display Full Details about Active Software Versions

From the CLI Maintenance/ManagedObjects level, type get versions current. The
following response appears (sample provided).
NS_CLI/Maintenance/ManagedObjects> get versions current
NS version Rel_17.0_1.444
Built Sat Apr 3 00:21:03 EDT 2010

- BASE revision 139841
- NS revision 139841
Patching Info:
Active Patches: 0
NOTE: BroadWorks software version information can also be viewed from the UNIX prompt as
bwadmin using the bwshowver command. For a list of all applied patches, the bwshowver
–showPatch option can be used.
8.2 List Installed Software Versions
8.2.1 List Installed and Active Software Versions

From the CLI Maintenance/ManagedObjects level, type get versions all. The
following response appears (sample provided).
NS_CLI/Maintenance/ManagedObjects> get versions all
Identity Version Install Date Status
===============================================
NS 15.0_1.290 Apr 3, 2010 Installed
NS 17.0_1.444 Apr 3, 2010 Active
2 entries found.
* Applications:
Name Version Install Date Upgrade Mode
Description Status
================================================================================
================================
NSExecutionAndProvisioning 17.0_1.444 Apr 3, 2010 Automatic Provides

©
translations and routing. Active
NSPortal 17.0_1.444 Apr 3, 2010 Automatic
NSPortal web portal Active
WebContainer 17.0_1.444 Apr 3, 2010 Automatic
Web Container Active
3 entries found.
* Third Party Software:

Third Party Version Status
======================================
java xs_java_base active
java jdk1.6.0_06 installed
java jdk1.6.0_14 active
apache 2.2.6 installed
apache 2.2.10b active
timesten 7.0.4.0.0 installed
timesten 7.0.5.9.0 active
tomcat 6.0.18b active
tomcat 6.0.14 installed
9 entries found.
8.3 Switch Active Software Version

The action of switching the active software version results in an upgrade or a rollback.
This action is performed using the set command in the Maintenance/ManagedObjects
CLI level.
BroadSoft suggests that you lock the server before restarting and switching the software
version.
NOTE: Restarting a server results in that server refusing any new incoming requests until the
server restarts.
1) From the CLI Maintenance/ManagedObjects level, type lock to (traffic) lock the
server. This ensures the remaining service handles all new calls. The server
transitions from the Unlocked to Locking state before finally entering the Locked state.
the server administration state is Locked.
state. Use the lock force command from the CLI Maintenance/ManagedObjects level to force
3) From the CLI Maintenance/ManagedObjects level, type set

activeSoftwareVersion server NS 17.0_1.444. The following response
NS_CLI/Maintenance/ManagedObjects> set activeSoftwareVersion server NS
17.0_1.444

This command will change the active software version of NS to
17.0_1.444. NOTE that this action will cause downtime.
Continue?

©
Please confirm (Yes, Y, No, N):
NOTE: Additional logs for the server restart and software version switch are written to file
/var/broadworks/logs/maintenance/setactiveserver_XXXXXXXXXXXX.log.
4) Exit the CLI after the activation of the new BroadWorks software version. This is
mandatory before issuing new CLI commands.

©
9 Scheduled Maintenance Tasks
There are certain tasks that can be scheduled on all BroadWorks servers. These tasks
are listed in the following tables.
Applicable To
Schedule
Access Application Database Element Execution Media
Tasks Mediation Server Server Management Server Server
Server System
AutoCleanup Weekly Weekly Weekly Weekly Weekly Weekly
Healthmon 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes
Backup Weekly Weekly Weekly Weekly Weekly Weekly
DbMaint N/A Weekly N/A Weekly N/A N/A
RegAudit N/A Optional N/A N/A N/A N/A
DbSyncCheck Not applicable Daily N/A N/A N/A N/A
CpuMon 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes
Tech-support Daily Daily Daily Daily Daily Daily
Check_dbpages N/A Daily N/A N/A N/A N/A
Service License N/A N/A N/A N/A

N/A Optional
Collect
File Collector Optional Optional N/A Optional Optional N/A
Applicable To
Schedule Network Profile Service Xtended Messaging Sharing WebRTC
Tasks Server Server Control Services Server Server Server
Function Platform
Server
AutoCleanup Weekly Weekly Weekly Weekly Weekly Weekly Weekly
Healthmon 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes
Backup Weekly Weekly Weekly Weekly Weekly Weekly Weekly
DbMaint Weekly N/A N/A N/A Weekly N/A N/A
RegAudit N/A N/A N/A N/A N/A N/A N/A
DbSyncCheck Daily N/A N/A N/A Daily N/A N/A
CpuMon 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes
Tech-support Daily Daily Daily Daily Daily Daily Daily
Check_dbpages Daily N/A N/A N/A Daily N/A N/A
Service License N/A N/A N/A N/A N/A N/A N/A

Collect
File Collector Optional Optional Optional Optional Optional Optional Optional

©
You can schedule tasks to run on a specific day of the month, a specific day of the week,
or every day at some specific hour or frequency.
AS_CLI/Maintenance/Scheduler> h add
The ADD command is used to add a new scheduled task.
A task can be configured to be executed on a minute, day or date basis.

Any given task can have multiple scheduled entries.
Following are examples:
"taskx minute 2" means execute taskx every 2 minutes
"taskx day monday 0 30" means execute taskx every monday at 0h30
"taskx date 15 2 15" means execute taskx on the 15 of each month at
2h15
======================================================================
add
<tasks>, Choice = { dbMaint, healthmon, backup, autoCleanup,
regAudit, dbSyncCheck, cpuMon, tech-support, check_dbpages,
serviceLicenseCollect, fileCollector }
<frequency>, Choice = {minute, daily, day, date}
minute:
<value>, Choice = {1, 2, 3, 4, 5, 6, 10, 12, 15, 30, 60, 120,
180…1440}
daily:
<hour>, Integer {0 to 23}
<minute>, Integer {0 to 59}
day:
<day>, Choice = {monday, tuesday, wednesday, thursday, friday,
saturday, sunday}
date:
<date>, Integer {1 to 28}
WARNING: Tasks should not be scheduled to run at the same time. Running all tasks at the
same time may affect server performance.
NOTE 1: Tasks should set to run at off peak hours.
NOTE 2: To minimize the impact of running concurrent scheduled tasks, it is recommended to

use minutes that are different between tasks, and those that do not match the hour (when
applicable).
NOTE 3: All maintenance tasks produce a daily log file located under /var/broadworks/logs/cron.

©
9.1 Automatic Backups
You use an automatic backup to schedule the backup of the TimesTen database and
other relevant file system directories. The backups are listed in the following tables.
Server
Description Access Application Database Element Execution Media

Mediation Server Server Management Server Server
Server System
Database X X X
/var/broadworks/
X
userfiles
/usr/local/broad
works/bw_base/ X X X X X X
conf/*
/var/TimesTen/
X
sys.odbc.ini*
Web
applications
Server
Network Profile Service Xtended Messaging Sharing WebRTC

Description
Server Server Control Services Server Server Server
Function Platform
Server
Database X X
/var/broadworks/
userfiles
/usr/local/broad
works/bw_base/ X X X X X X X
conf/*
/var/TimesTen/
X X
sys.odbc.ini*
Web applications X X
9.1.1 Mode of Operation

Execute backups as a UNIX cron job. Schedules can be controlled either with the
bwAutoBackup.sh script or from the CLI. The content of what is to be backed up on each
server is defined in /usr/local/broadworks/bw_base/conf/bwSystemBackup.conf.
At the scheduled time, the server creates a backup of all relevant information, compresses
it into one file, and places it in the following directory.
/usr/local/broadworks/bw_base/persistent/backups
The script deletes all old backup files. Backups should be transferred off the server for
safekeeping.
NOTE 1: Schedule backups to run on off-peak hours.

©
NOTE 2: Stagger backups across cluster peer members so that they do not run at the same
time on the primary and secondary Application Servers.
To configure automatic backups to run once a day at 3:30 A.M., from the CLI
Maintenance/Scheduler level, type add backup daily 3 30.
9.1.2 Auto-Backup File Naming Convention

An automatic backup file is a gzipped tar file called AutoBackupFileName, which is a
long name formatted as follows:
<YYYYdddHHmmSS>-<YYYYMMDD>-<random integer>-<hostname>-<hostid>
… where:
YYYY is the year
ddd is the day of year
HH is the hour of day
mm is the minute
SS is the second
MM is the month
DD is the day of month
Similarly, the database backup in the automatic backup file is called

DatabaseBackupFileName, which is a long name formatted as follows.
<YYYYdddHHmmSS>-<YYYYMMDD>-<random integer>-<hostname>-<hostid>-database
… where:
YYYY is the year
ddd is the day of year
HH is the hour of day
mm is the minute
SS is the second
MM is the month
DD is the day of month
9.1.3 Restore Auto-Backup

All files stored in the backup image are stored using relative paths. You can restore all
files or specific files using the following procedure.
Backup files only contain user data and general configuration files, not the entire contents
of the BroadWorks installation directory. If the server host to be restored no longer has
BroadWorks installed, you must first re-install the appropriate BroadWorks components,
and re-apply all BroadWorks patches that were in place when the backup was made.
Continue with the following procedure to restore the backup files.
Because the backup .tar file stores the relative paths of all included files, you need to
extract the archive (restore the server) from an appropriate directory. If you want to
restore files to the directory in which they were originally located when the backup was
made, perform the restore from the system root directory (/). Otherwise, the backup files
are simply extracted relative to the directory from which you run the tar xvf command.

©
Note that the restore operation should be executed as root when performed from the
system root directory (/).
1) The image must first be uncompressed using the gzip utility from the unzip prompt:
$ gunzip <backupName>.tar.gz
2) The content of the backup image can be viewed using the tar utility:
$ tar tvf <backupName>.tar
3) Stop BroadWorks, the SNMP Agent, and the Configuration Agent:
$ stopbw
$ snmpdctl stop
$ configdctl stop
4) Switch to the root account when restoring from the system root directory (/).
5) A specific file can be extracted by executing the tar command with the x option and by
specifying a specific file name (long name as stored in the backup):
6) $ tar xvf <backupName>.tar <fileToExtract>
Start the Configuration Agent, the SNMP Agent, and BroadWorks:
$ configdctl start
$ snmpdctl start
$ startbw
Example:
bwadmin@MTLNS01$ gunzip 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

MTLNS01-80fcc530.tar.gz
bwadmin@MTLNS01$ ls
2006010082501-20060110-5294-NS_Rel_14.0_1.240-MTLNS01-80fcc530.tar
bwadmin@MTLNS01$ tar tvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

MTLNS01-80fcc530.tar
bwadmin@MTLNS01$ tar tvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

MTLNS01-80fcc530.tar | grep persistent
-rw-rw-r-- 118/100 7804344 Jan 10 08:25 2006
usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database
bwadmin@MTLNS01$ stopbw
bwadmin@MTLNS01$ snmpdctl stop
bwadmin@MTLNS01$ configdctl stop
bwadmin@MTLNS01$ tar xvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

MTLNS01-80fcc530.tar
usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database
x usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database, 7804344 bytes, 15243 tape blocks
bwadmin@MTLNS01$ ls
2006010082501-20060110-5294-NS_Rel_14.0_1.240-MTLNS01-80fcc530.tar
usr/

©
bwadmin@MTLNS01$ ls -R usr
usr:
local/
usr/local:
broadworks/
usr/local/broadworks:
bw_base/
usr/local/broadworks/bw_base:
persistent/
usr/local/broadworks/bw_base/persistent:
backups/
usr/local/broadworks/bw_base/persistent/backups:
2006010082501-20060110-11469-MTLNS01-80fcc530-database
bwadmin@MTLNS01$ configdctl start
bwadmin@MTLNS01$ snmpdctl start
bwadmin@MTLNS01$ startbw
9.2 Automatic Disk Cleanup

Use automatic disk cleanup to schedule the automatic cleanup of specific directories on
the server to help manage disk space usage.
Directories Cleaned Up Action
/tmp All files older than two (2) days are deleted.
/var/tomcat/logs All files older than thirty (30) days are deleted.
/var/broadworks/logs/* All files older than thirty (30) days are deleted.
All files older than two (2) days are compressed.
NOTE: For the directory /var/broadworks/logs/*, the action applies to all directories under
/var/broadworks/logs.
Backups are executed as a UNIX cron job. Control the schedule with the
bwAutoCleanup.sh script or from the CLI.
For example, to set the cleanup to run once a week on Sunday at 4:30 A.M., from the CLI
Maintenance/Scheduler level, type add autoCleanup day sunday 4 30.
When the AutoCleanup task is run on a Database Server, the following files for Oracle
logs, traces, and alerts under the /u01/app/oracle/diag directory structure are compressed
(in addition to the log files under the /var/broadworks/logs directory):
 Audit Dump
 Database Instance traces
 ASM Instance traces

©
 TNS Listener traces
 TNS Listener alerts
 SCAN Listener traces
 SCAN Listener alerts
9.3 Healthmon Keep Alives

The healthmon tool is available on all BroadWorks servers. You use this tool to report the
list of processes in trouble, if any, and monitor key system components. For the
Application Server and Network Server, healthmon also monitors the state of the database
and redundancy. Database out-of-sync condition detection, database replication lagging
detection, and database notification lagging detection are part of the healthmon command.
Configure healthmon to run periodically with the SNMP Keep Alive functionality. The
severity of the trap is based on the state of the system. If failure conditions are not
detected, trap severity is a Notification. Invoke this mechanism through a Solaris cron job.
To configure healthmon to run every 15 minutes, from the CLI Maintenance/Scheduler
level, type add healthmon minute 15.
9.4 Database Statistics Update

The bwPeriodMaint.sh script is configured to run once a day at 2:15 A.M. This script
optimizes database performance and it should not be disabled. The time the script runs
can be altered to off-peak network hours.
WARNING 1: If you disable the bwPeriodmaint.sh task from the scheduler, the database may
exhibit performance issues.
WARNING 2: When the dbUpdateStats.sh script is running, it may prevent provisioning from
working. Make sure the script is scheduled to run outside of the busy hours.
WARNING 3: The time it takes to execute the dbUpdateStats.sh depends on the size of the
database and the hardware on which the application works. The execution can take a few
seconds up to a few minutes.
WARNING 4: This task also runs the check_dbpages task; therefore, both tasks should not be
scheduled at the same time.
9.5 Registration Audit

The RegAudit task is only available on the Application Server. It is responsible for auditing
the SIP registrations in the BroadWorks database. It automatically cleans up SIP
registrations, which have expired for more than a pre-defined period (configured through
the CLI extensionTimeInSeconds option under AS_CLI/System/ Registration).

©
9.6 Database Synchronization Check
The DbSyncCheck task is available on the Application Server and Network Server. It is
responsible for running the syncheck_basic.pl tool, which provides a lightweight
mechanism to determine if two databases are synchronized. For more details, see
section 11.1 Application Server, Network Server, and Messaging Server Database
Replication.
9.7 CPU Monitoring Tool

The CpuMon tool is available on all BroadWorks servers. CpuMon is responsible for
monitoring CPU idle percentage, and memory usage. It automatically generates an
SNMP trap with a severity level dependent on the configurable idle time threshold that has
been surpassed. This script uses either SAR (if installed) or vmstat over a one-minute
period.
9.8 Check DB Pages

The check_dbpages.pl tool is available on the Application Server and the Network Server.
This tool calculates the page size, detects pages that have exceeded their capacity, and
increases the capacity of the tables when needed. This ensures that provisioning
activities are not impacted by performance degradation. It should be configured to run
daily.
9.9 Tech-support
Run the tech-support maintenance task on a daily basis. The output of the tech-support
daily task is stored in the /var/broadworks/logs/monitoring directory.
9.10 Service License Collect

The ServiceLicenseCollect task gathers information about the current licensing and actual
usage of the various BroadWorks services. This information is stored in the:
/var/broadworks/serviceLicenseReporting/serviceLicenseCollect_(host)_(date).zip file.
This file is interpreted by the Service License Report Tool installed by default on the
Application Server.
9.11 Delete Scheduled Tasks

Scheduled tasks cannot be modified. To change the time that a task is configured to run,
delete the task, and then add a new task with the new time.
To delete a scheduled task:
1) From the CLI Maintenance/Scheduler level, type get. The following response
Id Name Date Day Hour Minute

===========================================================
1 healthmon - - - 15
2 dbMaint - sunday 2 33
3 backup - - 4 30
2) Type del <task ID>. The following response appears (sample provided).
...Done

©
10 Database Maintenance
BroadWorks uses TimesTen on the Application Server, Network Server, and Messaging
Server, MySQL on the Element Management System, and Oracle on the Database
Server.
All databases are delivered with Structured Query Language (SQL) clients and a set of
maintenance tools. In general, BroadSoft recommends the use of the commands
described in this document. All database-related issues or questions should be directed to
the BroadSoft Support line.
Maintenance procedures for Oracle are slightly different from those used for TimesTen
and MySQL. For more information on Database Server maintenance procedures, see the
BroadWorks Database Server Configuration Guide.
10.1 Database Backup

Backups can be performed with no impact to the system (for example, it is not required to
stop the BroadWorks application).
10.1.1 Manual Database Backup

1) Create a directory where the backups are stored. Example:
$ mkdir /var/dbbackup/Rel20.0/
2) To run the backup, type $ bwBackup.pl <dsn> <file>. DSN is the datastore
name (for example, NetworkServer, AppServer, ElementManagementSystem) and
file is the file name for the backup (including the path). Example:
Hostname$ bwBackup.pl NetworkServer /var/dbbackup/Rel20.0/Nsbackup.bak
The database can also be manually backed up from the CLI Maintenance/Tools level.
The DSN (datastore name) is not required.
NS_CLI/Maintenance/Tools> ?
0) backupdb : Backups the content of the BroadWorks TimesTen datastore
1) healthmon : Reports the status of the system
2) importdb : Imports the content of a remote TimesTen database
3) restoredb : Restores the content of the BroadWorks TimesTen datastore
4) tech-support : Captures the status of the system for troubleshooting
5) upgradeCheck : Performs pre-upgrade validation

c (config), cd (cd), a (alias), hi (history), p (pause), re (repeat)
NS_CLI/Maintenance/Tools> backupdb /var/broadworks/backup/backup9.bak

bwBackup.pl version 15.0.0
->Validating specified parameters... [done]
->Backing up dsn NetworkServer to /var/broadworks/backup/backup9.bak... [
in progress]
->Ensuring database is loaded in memory... [done]
->Backing up... [done]

©
10.1.2 Backup Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, or Messaging Server
1) Create a backup directory.
2) From the UNIX prompt (or from the CLI Maintenance/Tools level), type
bwBackup.pl.
3) Transfer the backup to some other storage facility for safekeeping.
10.2 Database Restore
WARNING: This procedure overwrites the Application Server, Element Management System,
Network Server, or Messaging Server database with the backed-up version. The existing data is
lost.
To restore the database, an existing backup must be present on the server. The restore
fails if there are processes attached to the database. Such processes can include
BroadWorks, replication daemons, and TimesTen (ttIsql) or MySQL database client tools.
10.2.1 Restore Database

Type Hostname$ bwRestore.pl <dsn> <file>. DSN is the datastore name and
where the file is the file name of the backup (including the path). The database can also
be restored from the CLI Maintenance/Tools level. DSN (datastore name) is not required.

NS_CLI/Maintenance/Tools> restoredb /var/broadworks/backup/backup9.bak

bwRestore.pl version 15.0.0
WARNING: ALL information in data store NetworkServer will be lost.
->Restoring dsn NetworkServer from /var/broadworks/backup/backup9.bak... [

in progress]
->Unloading database from memory... [done]
->Destroying store... [done]
->Restoring database from backup file... [done]
->Loading database in memory... [done]
->Updating statistics... [done]

©
10.2.2 Restore Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, or Messaging Server
1) From the CLI Maintenance/ManagedObjects level, type lock. This ensures that the
remaining service handles new calls. The server transitions from the Unlocked to
Locking state before it enters the Locked state.
the server administration state is Locked.
state. Use the lock force command form the CLI Maintenance/ManagedObjects level to force
3) From the UNIX prompt, type $ stopbw to stop the BroadWorks server.
4) If the server is part of a redundant cluster, stop the TimesTen replication (from the
UNIX prompt, type $ repctl stop).
5) Restore the desired database (using the bwRestore.pl command from the UNIX
prompt or from the CLI Maintenance/Tools level).
Example:
$ bwRestore.pl NetworkServer /var/dbbackup/Rel10.0/backup06.bak
6) If the server is part of a redundant cluster, restart TimesTen replication (from the UNIX
prompt, type $ repctl start).
7) Restart the BroadWorks Server (from the UNIX prompt type $ startbw).
NOTE: For a redundant cluster, restoring a database on just one peer results in a database out-
of-sync condition. A database import is required on the other peer to make sure a return to
database in-sync state.
10.2.3 Extract Database Backup from Automatic Backup File

1) Go to the directory with the automatic backup files. By default, this directory is
/usr/local/broadworks/bw_base/persistent/backups. The following steps assume this
is the directory where the automatic backup files are stored.
Identify the automatic backup file to extract the database backup. An automatic
backup file is a gzipped tar file called AutoBackupFileName, which is a long name
formatted as described in section 9.1.2 Auto-Backup File Naming Convention.
2) Unzip the automatic backup file. From the UNIX prompt, type $ gunzip
AutoBackupFileName.tar.gz.
3) The resulting tar file contains all the files that were automatically backed up. If -
database backup was enabled, the tar file contains the database backup file. The tar
file can also contain other files that had to be backed up. To identify the database
backup file from all the other files in the tar, type the following from the UNIX prompt.
$ tar tvf AutoBackupFileName.tar

©
If present, the database backup file (ending with -database) is the last file in the list.
Its full path name would be as follows.
$ /usr/local/broadworks/bw_base/persistent/backups/DatabaseBackupFileName
4) If you want to restore the backup files to their original directories, change your working
directory to the system root directory (/). Otherwise, the backup files are simply
extracted relative to the directory from which you run the tar xvf command. Type:
$ cd /
5) To extract the database backup file from the automatic backup tar file, from the UNIX
prompt, type:
$ tar xvfP AutoBackupFileName.tar
/usr/local/broadworks/bw_base/persistent/backups/DatabaseBackupFileName
6) The database backup should now be in the directory, ready to be restored by

following the instructions in section 10.2 Database Restore.
10.3 Database Import
WARNING: This procedure overwrites the Application Server, Element Management System,
Network Server, or Messaging Server database with the backed-up version. The existing data is
lost.
A database can be imported from a cluster peer member. This can be used to make sure
that the databases across all peers are in-sync.
The import fails if there are processes attached to the database. Such processes can
include BroadWorks, replication daemons, and TimesTen (ttIsql) or MySQL client
tools. The import requires that the TimesTen or MySQL replication daemon is running on
the peer server.
The following command can be used to import a database.
$ importdb.pl [-noWarning] <local dsn> <remote host address> <remote dsn>
… where the local/remote DSN is the datastore name (AppServer,

ElementManagementSystem, NetworkServer, or UMS) and the remote host address is
the address of the peer server from which you are importing (as set in the peerctl ls).
For example, if you are importing from as1 to as2 and the peerctl ls is as follows:
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
as1/192.168.13.102 primary, unlocked
as2/192.168.13.103 unlocked
...then the remote host name should be set to “192.168.13.102”. It is always the peer’s
peerctl address.
NOTE: The loghost must be associated with this peer in the /etc/hosts file.

©
The database can also be restored from the CLI Maintenance/Tools level. DSN
(datastore name) is not required.

NS_CLI/Maintenance/Tools> importdb mtl64lin17

-> Making safety backup <-
Backing up dsn NetworkServer to /var/broadworks/backup/NS_Rel_14.0_1.538.0607
14160846.backup... [done]
-> Importing data from primary server <-
Retrieving localhost in remote replication schema... [mtl64lin10.mtl.broadsof
t.com]
Performing checkpoint on remote database... [done]
Unloading database from memory... [done]
Destroying database NetworkServer... [done]
Importing the remote database... [done]
10.3.1 Import Database on Application Server, Element Management System, Network

Server, or Messaging Server
1) Traffic lock the server (from the CLI, go to Maintenance/ManagedObjects and type
lock). This makes sure that all new calls are handled by the remaining in service
cluster peer.
2) The server transitions from the Unlocked to Locking state before finally entering the
Locked state. Before proceeding, from the CLI Maintenance/ManagedObjects level,
verify that the server administrative state is Locked by typing get broadworks.
state. Use the lock force command form the CLI Maintenance/ManagedObjects level to
force the administration state to Locked.
3) Stop the BroadWorks Server (from the UNIX prompt type $ stopbw).
4) Lock the database on the server (from the UNIX prompt type $ peerctl lock).
5) Stop TimesTen replication, if the server is part of a redundant cluster (from the UNIX
prompt type $ repctl stop).
6) Import the database from the cluster peer (using the importdb.pl command from the
UNIX prompt or from the CLI Maintenance/Tools level). Example:
$ importdb.pl NetworkServer myhost.mycompany.com NetworkServer
7) Restart TimesTen replication, if the server is part of a redundant cluster (from the
UNIX prompt, type $ repctl start).
8) Unlock the database on the server (from the UNIX prompt type $ peerctl
unlock).
9) Restart the BroadWorks server (from the UNIX prompt, type $ startbw).

©
10) Unlock traffic on server (from the CLI, go to Maintenance/ManagedObjects and type
unlock).
10.4 Database Installation
DANGER: This procedure destroys the existing Application Server, Element Management
System, Network Server, or Messaging Server database and installs a fresh database. This
command should NEVER be run on a production server unless under the direct supervision of
BroadSoft Support.
NOTE: For a redundant cluster, restoring a database on just one peer results in a database
out-of-sync condition. A database import is required on the other peer to ensure a return to
10.4.1 Recreate Application Server, Network Server, or Messaging Server Database

It is possible to force an installation or reinstallation of the TimesTen datastore. The first
option is to force an installation when an upgrade was chosen during a BroadWorks
installation. This destroys the current datastore and it installs a fresh, empty data schema,
which can be useful in a lab environment.
 Network Server: Hostname$ dbsetup.sh NS_Rel_18.0_1.xxx –install
 Application Server: Hostname$ dbsetup.sh AS_Rel_18.0_1.xxx –install
 Messaging Server: Hostname$ dbsetup.sh UMS_Rel_20.0_1.xxx –install
If you only need to revert to the datastore as it was right after the installation, or if it was not
installed, you would issue the following commands:
 Network Server: Hostname$ dbsetup.sh NS_Rel_16.0_1.xxx –install
 Application Server: Hostname$ dbsetup.sh AS_Rel_16.0_1.xxx –install
 Messaging Server: Hostname$ dbsetup.sh UMS_Rel_20.0_1.xxx –install
A fresh install fails when there are processes attached to the database. Such processes
can include BroadWorks, replication daemons, and TimesTen tools, such as ttIsql.
bwadmin@ns1$ dbsetup.sh NS_Rel_16.0_1.500 -install
---- BroadWorks database setup utility version 16.0 ----
=> Gathering data <=

in progress [....
********************************************************************
*
WARNING - WARNING - WARNING - WARNING - WARNING - WARNING
This command can only be run when the server is in Maintenance

state.
The dbsetup.sh utility is meant to be invoked by internal

BroadWorks commands. It should be used with caution since it could
cause a loss of all data contained in the server database.
Please refer to BroadWorks personnel before directly invoking that

that command.

©
********************************************************************
*
........]
=> Checking for perm size usage <=
Perm Size Usage = 6% (limit is at 55%)
=> Checking for available disk space <=
Perm = 128 Temp = 42
Space required [16 Mb]
Checking on /bw... [done]
=> Creating bw_base link... [done]
=> Settings <=
Target BroadWorks version [NS_Rel_16.0_1.500]
Data store name [networkserver]
TimesTen version [7.0.5.9.0]
Force installation [true]
Installation type [fresh install]
Redundant System [false]
Current maintenance... [NS_Rel_16.0_1.500]
=> Installing the latest data schema <=
Creating tables... [done]
Loading basic data... [done]
=> Now configuring replication <=
Adding ns1/ns1... [done]
Setting primary peer (ns1)... [done]
Setting replication port (17888)... [done]
=> Setting IMS mode options <=
setting IMS mode to false... [done]
=> Setting platform mode options <=
=> Checking crontab for DB periodic maintenance <=
Crontab entry already present
=> Verifying database paging <=
=> Ensuring database is loaded in memory <=
Current maintenance... [NS_Rel_16.0_1.500]
Setting ram policy to always [done]
==== BroadWorks database setup [DONE] ====
10.4.1.1 Reinstall Database on Network Server, Application Server, or Messaging Server

1) Stop the BroadWorks Server (from the UNIX prompt type $ stopbw).
2) Stop TimesTen replication if the server is part of a redundant cluster (from the UNIX
prompt, type $ repctl stop).
3) Either reinstall the database, a fresh install or revert (using the dbsetup.sh
command from the UNIX prompt). For example:
$ ./dbsetup.sh NS_Rel_16.0_1.500 –install.
4) On the Network Server only, reinstall the desired dial plans (from the UNIX prompt,
type installdialplan.sh <NS version>).
5) If the server is part of a redundant cluster, then restart TimesTen replication (from the
UNIX prompt, type $ repctl start).
6) Restart the BroadWorks server (from the UNIX prompt, type $ startbw).

©
10.5 Additional TimesTen Maintenance Tasks
10.5.1 Change TimesTen Database Size
WARNING 1: This procedure can be used to increase or decrease the database size and
should only be run with the assistance of BroadSoft Support personnel.
WARNING 2: During this procedure, the servers are locked, preventing provisioning changes
and device registration.
An operator shall use the resizeDSN tool packaged with the BroadWorks server to change
the database size. This following procedure provides the steps required to change the
database store name (DSN) size on an Application Server, a Network Server, or a
Messaging Server cluster. For safety, make a backup of the database before starting the
procedure. Always start the procedure on the primary server and then execute it on the
secondary servers. The procedure must be executed on one server at a time.
The following step must ONLY be run on the primary server:
1) Lock all servers in the cluster ($peerctl lock all).
The remainder of the procedure must be executed on the primary server first. Once
completed on the primary server, the remainder of the procedure must be executed on all
servers in the cluster, one at a time:
2) Stop BroadWorks on the server ($stopbw).
3) Stop replication on the server ($repctl stop).
4) Unload the database from memory ($timesten.pl unload).
5) Log in as root ($su).
6) Go to the BroadWorks bin directory (#cd
/usr/local/broadworks/bw_base/bin).
7) Run the resizeDSN command (#./resizeDSN).
During the execution of the resizeDSN command, when prompted with the question
“Do you want to back up the database (optional when increasing the DSN size) (y/n)
[y]?”, it is recommended to answer “n” (no) when the DSN size is increased. When
you answer no, the DSN is resized and then the TimesTen database is loaded. When
you answer yes, a backup is made, followed by a question about whether to restore
the safety backup, which adds unnecessary time to the activity.
8) Come back as bwadmin (#exit).
9) Restart replication on the server ($repctl start).
10) Unlock the server ($peerctl unlock).
11) Restart BroadWorks on the server ($startbw).

©
10.5.2 Update TimesTen Database Statistics
NOTE: This procedure automatically runs weekly and should not run manually unless under the
supervision of BroadSoft Support personnel. Running this procedure can affect performance.
To set the database to update statistics:

Run /usr/local/broadworks/bw_base/bwPeriodMaint.sh.
10.5.3 Start and Stop TimesTen Database Daemon
WARNING: The TimesTen database is started automatically upon server power up or reboot.
The database should not be manually stopped unless under direct supervision from BroadSoft
Support.
The TimesTen database daemon is controlled through the following command.

/etc/init.d/tt_daemon[stop | start]
The database cannot be stopped if processes are attached to the database. Such
processes can include BroadWorks, replication daemons, and TimesTen tools like
ttIsql.
To stop the TimesTen database daemon, run the following commands:
1) Stop the BroadWorks server (from the UNIX prompt type $ stopbw).
2) Stop TimesTen replication if server is part of a redundant cluster (from the UNIX
prompt type $ repctl stop).
3) Log in (su -) as root and run the following commands:
# cd /etc/init.d/
# ./tt daemon stop
10.5.4 Update TimesTen Database Paging Information

The database paging information must be re-calculated and re-optimized occasionally.
When the paging is not properly set on a server, major performance degradations may
occur.
The healthmon command periodically runs the check_dbpages.pl script using the detect
option to verify that the paging information is properly set. Should the check_dbpages.pl
script report an issue, an operator may have to run the same command with the modify
option to perform an optimization. The check_dbpages.pl script modifies only the
Application Server on the current host; therefore, it should be run against each redundant
Application Server that reports the same issue.
NOTE: It is recommended to run the check_dbpages.pl with the modify option during hours of
low traffic volume, as it temporarily locks the database.

©
10.5.5 Compact the Database
Over time (after months of operation), it is possible that data in the database becomes
fragmented causing the PERM size in use to be more than it should. This can also have
an impact on the performance of the database. When the datastore in use becomes close
to the limit of the DSN, it is either recommended to try to compress it (increasing the
datastore may not be possible because there is restriction on the DSN size base on the
available RAM on a server). The following is the procedure for performing the
consolidation of the TimesTen database. Note that the primary server must always be
done first.
10.5.5.1 Compact the Primary Server

1) Stop BroadWorks ($stopbw).
2) Lock all peers ($peerctl lock all).
3) Stop replication on all peers ($peercmd “repctl stop”).
4) Back up the database ($bwBackup.pl <dsn> <backup file> -migrate).
5) Restore the database ($bwRestore.pl <dsn> <backup file> -migrate).
6) Restart replication ($repctl start).
7) Unlock the primary server ($peerctl unlock).
8) Restart BroadWorks ($startbw).
10.5.5.2 Compact Backup Servers

1) Stop BroadWorks ($stopbw).
2) Import the database from the primary server ($importdb.pl).
3) Restart replication ($repctl start).
4) Unlock the server ($peerctl unlock).
5) Restart BroadWorks ($startbw).
10.5.6 Import TimesTen Datastores between Different Operating Systems

This section provides a procedure to migrate the Application Server and Network Server
database from Linux to Solaris or Solaris to Linux. Since the images obtained through
bwBackup.pl are not compatible between the Solaris and Linux operating systems, one
way to bypass the problem is by using a remote connection of the source database from
the destination datastore.
SOURCE_NODE = This is the box you are copying from.
TARGET_NODE = This is the box you are copying to.
NOTE 1: This procedure assumes that BroadWorks is installed on the TARGET_NODE. It is

important that both the SOURCE_NODE and the TARGET_NODE are at the exact same
release and patch level.
NOTE 2: There is no way to take a backup image obtained through bwBackup.pl and restore it
directly on a server that has an operating system different from that of the server from which the
backup was obtained.

©
1) Stop BroadWorks where TARGET_NODE is running (from the UNIX prompt type
stopbw).
2) Stop BroadWorks on the SOURCE_NODE (from the UNIX prompt type stopbw).
3) Stop the replication on the SOURCE_NODE (from the UNIX prompt type repctl
stop).
4) From the TARGET_NODE, execute the following command.
ttMigrateCS -c -v0 –connStr "TTC_Server=<SOURCE_NODE>;
TTC_Server_DSN=<DSN>; TCP_Port=<PORT>;UID=bw;PWD=bw"
/var/broadworks/backup/remote_db_backup.ttm
Where
SOURCE_NODE is the address of the source node
DSN is AppServer or NetworkServer depending on the BroadWorks
server type
PORT depends on the TT version, 16003 (can be obtained from
ttStatus)
5) Restore the backup on the TARGET_NODE.

bwRestore.pl <DSN> /var/broadworks/backup/remote_db_backup.ttm –migrate
6) Change the host name in the database using peerctl (from the UNIX prompt type
peerctl set).
7) In case the SOURCE_NODE was in a redundant configuration, from ttIsql on the
TARGET_NODE, drop the previous replication schema (from ttIsql, do drop
replication BW.REP;).
Once the procedure is completed, it is recommended to perform the following steps on the
SOURCE_NODE:
1) Import the database from its peer (from the UNIX prompt, type importdb.pl).
2) Start the replication (from the UNIX prompt, type repctl start).
3) Start BroadWorks (from the UNIX prompt, type startbw).
Following is a sample execution of the procedure described in this sub-section.
bwadmin@mtlsoldev1$ ttMigrateCS -c -v0 -connStr "TTC_Server=mtl64lin16;
TTC_Server_DSN=AppServer; TCP_Port=16003; UID=bw; PWD=bw"
/var/broadworks/backup/remote_db_backup.ttm
bwadmin@mtlsoldev1$ bwRestore.pl AppServer
/var/broadworks/backup/remote_db_backup.ttm -migrate
bwRestore.pl version 14.2.0
WARNING: ALL information in data store AppServer will be lost.

->Restoring from backup with migrate option...
Press ctrl-c now to abort or <enter> to continue...
->Restoring dsn AppServer from

/var/broadworks/backup/remote_db_backup.ttm... [in progress]
->Unloading database from memory... [done]
->Destroying store... [done]
->Restoring database from backup file... [done]
->Loading database in memory... [done]
->Updating statistics... [done]
bwadmin@mtlsoldev1$ peerctl ls

©
-----------------------------------------------------------------------
---------
mtl64lin16.mtl.broadsoft.com/mtl64lin16.mtl.broadsoft.com
primary,unlocked
mtl64lin01.mtl.broadsoft.com/mtl64lin01.mtl.broadsoft.com
unlocked
bwadmin@mtlsoldev1$ peerctl del mtl64lin01.mtl.broadsoft.com
peerbwadmin@mtlsoldev1$ peerctl set mtl64lin16.mtl.broadsoft.com
mtlsoldev1
bwadmin@mtlsoldev1$ peerctl ls
-----------------------------------------------------------------------
---------
mtlsoldev1/mtlsoldev1 primary,unlocked
10.5.7 TimesTen Connection Optimization
NOTE: The following change is required on boxes with more than 32 CPUs, in other words, on
T2000 boxes. It should be implemented on these boxes across the board.
On certain servers that are large, the default number of connections provided to TimesTen
is not enough. This information can be retrieved by running the tech-support script and
looking at the TimesTen Server information and status section in the following example.
This section provides information about TimesTen including the number of connections.
Example:
TimesTen status report as of Sun Oct 16 10:05:20 2011
Daemon pid 1842 port 11284 instance 11.2.1.8.4

TimesTen server pid 1850 started on port 11285
------------------------------------------------------------------------
Data store /bw/broadworks/persistent/AppServer
There are 48 connections to the data store <<<<----- 48 connections
Shared Memory KEY 0x10036160 ID 557061
Type PID Context Connection Name
ConnID
Process 3087 0x000000001821aa90 remotexla 5
Process 3087 0x0000000018323440 remotexla 6
Process 3583 0x00000000468846d0 Execution Server 7
Process 3583 0x000000004698eb50 Execution Server 11
Process 3583 0x00000000469f81d0 Execution Server 12
[...]
In the example, TimesTen is using 48 connections. The default number of connections

provided to TimesTen is 256. If you reach that limit, you can increase the number of
connections by performing the following steps:

©
1) Edit the /var/TimesTen/sys.odbc.ini file and change all occurrences of
“Connections=256” to a higher value. For example, you can set a value of “300”.
2) To make this change effective, you must perform the following:
− stopbw
− repctl stop
− timesten.pl unload
− timesten.pl load
− repctl start
− startbw
If the system is redundant, then these steps must be done on all peers.
10.6 Additional MySQL Maintenance Tasks
10.6.1 Update MySQL Database Statistics
NOTE: This procedure automatically runs every night at 2:15 A.M. and should not be run
manually unless under the supervision of BroadSoft Support personnel. Running this procedure
can affect performance.
To set the database to update statistics:

Run /usr/local/broadworks/bw_base/bwPeriodMaint.sh.
10.6.2 Start and Stop MySQL Database Daemon
WARNING: The MySQL database is started automatically upon server power up or reboot. The
database should not be manually stopped unless under direct supervision from BroadSoft
Support.
The MySQL database daemon is controlled through the following command run as root.
service mysql_daemon [stop | start | restart]
The database cannot be stopped if processes are attached to the database. Such
processes can include BroadWorks, replication daemons, and MySQL tools such as
mysql.
To stop the MySQL database daemon, run the following commands:
1) Stop the BroadWorks server (from the UNIX prompt type $ stopbw).
2) Stop MySQL replication if server is part of a redundant cluster (from the UNIX prompt
type $ repctl stop).
3) Log in (su -) as root and run the following commands.
# service mysql_daemon stop

©
10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files
InnoDB stores data in a tablespace. By default on BroadWorks servers, data (data +
indexes) for each table are stored in a separate tablespace file. The tablespace file is
automatically extended as needed when the table grows (entries are inserted into the
table). When data is removed from the table (entries are deleted), the tablespace file size
does not decrease. This is because InnoDB cannot return space to the file system.
Although, it makes sure to use the freed-up space for further inserts. Perform the
following steps to reclaim space after the tables are purged:
1) Back up the database. For detailed steps, see section 10.1 Database Backup.
2) Restore the database. For detailed steps, see section 10.2 Database Restore.
NOTE: For a redundant cluster, restoring a database on just one peer results in a database out-
of-sync condition. A database import is required on the other peer to ensure a return to

©
11 Redundancy Management
Redundancy management for the Database Server is described in the BroadWorks

Database Server Configuration Guide. For more information on database redundancy
management, see this guide.
11.1 Application Server, Network Server, and Messaging Server Database

Replication
11.1.1 TimesTen Database Synchronization

The TimesTen replication daemon makes sure that database information across cluster
peers (Application Server, Network Server, and Messaging Server clusters) remains
synchronized. It starts automatically on a fresh install or upgrade and runs independently
of the BroadWorks application (that is, a stopbw does not stop replication). It is important
that replication be running on all cluster peers at all times.
WARNING: Stopping TimesTen Replication can result in cluster member databases becoming
out-of-sync. Replication should only be stopped when indicated in a BroadWorks maintenance
procedure or under the supervision of BroadSoft Support personnel.
In addition to an out-of-sync condition, failure to run replication on all cluster peers at all
times can possibly lead to longer re-synchronization times once replication is re-enabled
later. The Application Server, Network Server, and Messaging Server track three states
related to the database:
 Replication State – For each peer in the cluster, the state of whether file replication
and database replication are running is tracked.
 Database Replication Lagging State – The database retains transactions (and their
associated log files) if they have not been successfully replicated to all remote peers.
If a peer is shut down (entire server shut down, including replication) for maintenance
for an extended period and replication is not turned off on the operating server, a high
number of transaction files are retained by replication on the operating server. This
can cause excessive disk space usage and result in a long time for databases to re-
synchronize when the previously shut down server is re-instated. For each peer in the
cluster, the number of database transaction log files retained by replication is tracked.
Also tracked are whether an excessive number of transaction log files have been
retained by replication and whether an import resynchronizes the databases faster
than replication.
This threshold is defined at 90% of 360 transaction files (that is, 324 files).
 Database Notification Lagging State – The database retains transactions (and their
associated log files) if associated notifications have not been delivered to the
application. If database notifications have not been delivered to the application, or if
there is an excess of database notifications, a high number of transaction log files are
retained by database notification. This can cause excessive disk space usage,
resulting in the application cache not being synchronized with the database, and
points to a possible defect in the notification delivery mechanism. For each peer in
the cluster, the number of database transaction log files retained by notification is
tracked. Also tracked is whether an excessive number of transaction log files have
been retained by notification.

©
This threshold is defined at 20 * number of central processing units (CPUs). For
example, the threshold for one processor system is 20 transaction log files. The
threshold for a four-processor system is 80 transaction log files.
The healthmon command with the –d option outputs details for the three states. The
summary (true/false status) for the three states is output as part of invoking repctl status.
Warnings for replication lagging and notification lagging exceeding threshold are
generated by repctl stop, healthmon (without the -d option), and BroadWorks startup.
Replication can be stopped or started from the CLI System/Peering level or from the
UNIX prompt using the repctl command.
WARNING: BroadWorks must be stopped on the server prior to stopping and starting, or
restarting the replication. When the replication is restarted on the BroadWorks primary server
(the server hosting the most up to date database), all peers must be imported from that primary
server.
To stop the TimesTen replication daemon:

From the UNIX prompt, type $ repctl stop. The following response appears (sample
provided).
Locking local database. This may take a few minutes. . .
Database already locked
Pushing transactions held by replication to remote peers
Keeping the database locked
Shutting down Redundancy/Replication
Stopping file replication (this may take up to 600 seconds, please be
patient)
Database Redundancy:
Stopping Database Redundancy... [done]
Dropping Replication Schema... [done]
NOTE: As seen in the sample output above, repctl stop pushes transactions held by replication
to all peers in the cluster. If the transactions held by replication could not be pushed to all peers
in the cluster, the following warning is generated.
WARNING: Outstanding transactions could not be replicated to all remote peers. Databases
are out-of-synch. Run repctl status for more details on replication status. If required, import the
database from the appropriate cluster member to all other cluster members to synchronize the
system.
To start the TimesTen replication daemon:

From the UNIX prompt, type $ repctl start. The following response appears
(sample provided).
Checking replication status on peer servers:
-> Replication is clear for start up
Starting Replication:
Database Replication:
Performing Checkpoint... [done]
Creating Replication Schema... [done]
Starting Database Redundancy... [done]

©
NOTE 1: The replication daemon can be bounced (stopped/started) in one step using the
repctl restart option.
NOTE 2: repctl start checks the replication lagging state on all servers in the cluster.
If replication is found to be lagging on any server in the cluster, the following message is
displayed:
WARNING: Replication is lagging on one of the servers in the cluster. Invoke repctl status for
more details. Use database import to synchronize with the server exhibiting replication lagging.
11.1.2 TimesTen Replication Monitoring

Replication status and database notification status can be checked using the UNIX prompt
command repctl status. It is critical to the BroadWorks Redundancy solution that the
replication daemon is running at all times. Replication status can be checked using the
UNIX prompt command repctl status.
To view the TimesTen replication daemon state:
From the UNIX prompt, type $ repctl status. The following response appears
(sample provided).
Redundancy/Replication Status
-----------------------------
File Replication pid(s) = 1436
Datastore name = AppServer
Replication Agent Policy : always
Replication State
MTLAS04: (filerep: started) (database: started)
Database Replication Lagging State

MTLAS04: ok (1 files held)(earliest LSN 40/6688160)
Database Notification Lagging State

MTLAS04: ok (1 files held)
(ExecutionServer earliest LSN 40/6676296)
(ProvisioningServer earliest LSN 40/6700176)
The number after the File Replication PID field is the RSYNC PID number. If no number is
present, file replication is not running on the local server. The Replication Agent Policy
field provides information on the state of TimesTen Replication on the local server. A
value of “always” indicates the replication is running. A value of “never” indicates the
replication is not running.
The state of database replication and file replication for all peers in the cluster is output
under the title “Replication State” as shown in the example above. The entries are
prefixed by the canonical host name of each peer.

©
The state of database replication lagging for each peer in the cluster is output under the
title “Database Replication Lagging State” as shown in the example above either as “ok” or
“LAGGING !”. For each peer in the cluster, the canonical host name is output followed by
a status, indicating whether replication is lagging at that peer. The number of database
transaction log files held by replication is output, followed by the TimesTen log sequence
number (that is, LSN) of the earliest transaction held by the TimesTen replication daemon.
The LSN is of the format filenum/transactionNum. The filenum corresponds to the
transaction log file number for transaction logs in
/usr/local/broadworks/bw_base/persistent. The transactionNum is an internal transaction
number.
The state of database notification lagging for each peer in the cluster is output under the
title “Database Notification Lagging State” as shown in the example above. For each peer
in the cluster, the canonical host name is output followed by a status indicating whether
database notification is lagging at that peer. The number of database transaction log files
held by database notification is output followed by the TimesTen log sequence number of
the earliest transaction held by the TimesTen replication daemon. On the Application
Server, there are two entries related to the number of transaction log files held
(ExecutionServer and ProvisioningServer as shown in the preceeding output) for each
peer. On the Network Server, there are other entries for each peer.
To obtain more information about database replication lagging and database notification
lagging, use healthmon -l -d. See the following description of such details.
The healthmon script conducts a full two-way database replication test. A replication
failure results in the following error message.
--------------------------------
System Health Report Page
BroadWorks Server Name: as2
Date and time : Fri Jun 6 13:44:19 EDT 2003
Report severity : CRITICAL
Server type : AppServer
--------------------------------
Replication: a remote server is not receiving replication updates.
Databases may be out-of-synch.
--------------------------------
Recommendations
---------------
Replication must be started on all remote servers (repctl start). If

databases are out-of-synch, they must be re-synchronized first (with the
importdb.pl tool). Please refer to the BroadWorks Maintenance Guide for
detailed procedures.
--------------------------------
Healthmon also checks the database replication lagging state and database notification
lagging state. Lagging results in the following recommendations.
--------------------------------
--------------------------------

©
Recommendations
---------------
Replication is severely lagging on one of the servers in the cluster. If
the system is being operated in standalone mode, turn off replication.
Otherwise, import the database from the server exhibiting the lagging to
ensure prompt data synchronization.
Database XLA is severely lagging on one of the servers in the cluster.

If the files held by database notification continue to grow and the LSN’s
do not advance, contact BroadSoft support prior to restarting the
affected server.
--------------------------------
NOTE: Healthmon should be set up as a scheduled task running at least once an hour.
WARNING: If the repctl status indicates replication is not running, Healthmon indicates that a
remote server is not receiving replication updates, or repctl status/healthmon indicates that
database replication lagging exists at one of the peers in the cluster and the databases need to
be resynchronized. Database resynchronization is accomplished using the Import Database
procedure. In general, the import is run on the secondary server.
11.1.3 TimesTen Replication Port Number

It is possible to specify a port to be used by the TimesTen daemon for replication. By
default, the port number is assigned randomly.
NOTE: A change to a replication port only takes effect after the replication is bounced.
Therefore, changing the port MUST be done during a maintenance window.
To change a replication port, an operator must bounce replication. As such, to prevent

any data de-synchronization, it is important to adhere to the following rules and steps:
1) Change the replication port at the primary server in the cluster. Note that when you
change the replication port, it is automatically replicated to all replication peers.
NS_CLI/System/Peering> get
replicationPort = random
NS_CLI/System/Peering> set replicationPort port 19999

...Done
NS_CLI/System/Peering> get
replicationPort = 19999
2) From the primary server, lock all servers for provisioning.

bwadmin@MTLNS01$ peerctl lock all
3) From the primary server, stop replication on all peers.

bwadmin@MTLNS01$ peercmd repctl stop

©
4) On the primary server, restart replication.
bwadmin@MTLNS01$ repctl start
5) Unlock the provisioning “function” on the primary server.

bwadmin@MTLNS01$ peerctl unlock mtlns01
On the secondary server for the Application Server or each peer (one at a time) on the
Network Server, and Messaging Server, perform the steps according to the following
commands (6 through 10):
6) Stop BroadWorks.
bwadmin@MTLNS02$ stopbw
7) Import the database.

bwadmin@MTLNS02$ importdb.pl networkserver mtlns01 networkserver
8) Restart the replication.

bwadmin@MTLNS02$ repctl start
9) Unlock the provisioning “function”.

bwadmin@MTLNS02$ peerctl unlock mtlns02
10) Restart BroadWorks.

bwadmin@MTLNS02$ startbw
It is also possible to view the currently used replication port using bwListPortInUse
script as follows.
bwadmin@MTLNS01$ bwListPortInUse
Usage Type Port Internal
----------------------------------------------------------------
SNMP agent UDP 8001 N
SNMP agent trap server TCP 30002 Y
MO daemon TCP 7120 Y
MO root daemon TCP 7121 Y
Patch Tool TCP 2223 N
Patch Tool (Perl) TCP 2224 Y
TimesTen Main Daemon TCP 16003 N
Replication port TCP 19999 N
Naming service port TCP 1050 Y
SIP listening port UDP 5060 N
MSS listening port UDP 5677 N
ASR listening port UDP 5090 N

©
11.2 Element Management System Database Replication
11.2.1 MySQL Database Synchronization

The version MySQL packaged with BroadWorks supports for one-way replication. One
server acts as the master, while one or more other servers act as slaves. The master
server writes updates to its binary log files, and maintains an index of the files to keep
track of log rotation. These logs serve as a record of updates to be sent to slave servers.
When a slave server connects to the master server, it informs the master of its last
position within the logs since the last successfully propagated update. The slave catches
up on any updates that have occurred since then, and then blocks and waits for the
master to notify it of new updates.
Note that BroadWorks uses twice one-way master-slave replication to provide two-way
replication functionality. This implies that data written on any of the servers in a pair of
servers is replicated to its peer server.
The MySQL replication ensures that database information across cluster peers (Element
Management System pair) remains synchronized. It starts automatically on a fresh install
or upgrade, and runs independently of the BroadWorks application (that is, a stopbw
does not stop replication). It is important that replication be running on both servers of a
pair at all times.
WARNING: Stopping MySQL replication can result in cluster member databases becoming out-
of-sync. Replication should only be stopped when indicated in a BroadWorks maintenance
procedure or under the supervision of BroadSoft Support personnel.
In addition to an out-of-sync condition, failure to run replication on all cluster peers at all
times can possibly lead to longer re-synchronization times when replication is re-enabled
later. The Element Management System tracks the states of the master and slave
connection to the database. Under normal operation, the state is set to “Waiting for
master to send event”.
Replication can be stopped or started from the CLI System/Peering level or from the
UNIX prompt, using the repctl command.
To stop the MySQL replication daemon:
From the UNIX prompt, type $ repctl stop. The following response appears (sample
provided).
bwadmin@virems01.mtl.broadsoft.com$ repctl stop
File Replication:
Stopping File Replication... [done]
WARNING: Outstanding transactions could not be replicated to all remote peers. Databases
are out-of-synch. Run repctl status for more details on replication status. If required, import the
database from the appropriate cluster member to all other cluster members to synchronize the
system.

©
To start the MySQL replication daemon:
From the UNIX prompt, type $ repctl start. The following response appears
(sample provided).
bwadmin@virems01.mtl.broadsoft.com$ repctl start
Checking replication status on peer servers... [done]
virems02.mtl.broadsoft.com: Checking replication status... Started.
Versions must match.
virems02.mtl.broadsoft.com: Checking version... Versions matched (EMS
version Rel_16.0_1.483).
-> Replication is clear for start up
Starting File Replication... [done]
NOTE: The replication daemon can be bounced (stopped/started) in single step using the repctl
restart option.
11.2.2 MySQL Replication Monitoring

Replication status can be checked using the UNIX prompt command repctl status. It is
critical to the BroadWorks Redundancy solution that the replication daemon is running at
all times.
To view the MySQL replication daemon state:
From the UNIX prompt, type $ repctl status. The following response appears
(sample provided).
bwadmin@virems01.mtl.broadsoft.com$ repctl status
-----------------------------
Datastore name = WebNmsDB
Replication State
virems01.mtl.broadsoft.com: (filerep: true)(database: true)
virems02.mtl.broadsoft.com: (filerep: true)(database: true)

virems01.mtl.broadsoft.com: (false) Master/Slave position:
3993266/3993266 (delta: 0)
virems02.mtl.broadsoft.com: (false) Master/Slave position:
4418021/4418021 (delta: 0)
The number after the File Replication PID field is the RSYNC PID number. If no number is
present, file replication is not running on the local server. The Replication Agent Policy
field provides information on the state of MySQL Replication on the local server. A value
of “always” indicates the replication is running. A value of “never” indicates the replication
is not running.
The state of database replication and file replication for all peers in the cluster is output
under the title “Replication State” as shown in the example above. The entries are
prefixed by the canonical host name of each peer.

©
The state of database replication lagging for each peer in the cluster is output under the
title “Database Replication Lagging State” as shown in the example above either as
“false”, that is, no lagging, or “true”. For each peer in the cluster, the canonical host name
is output followed by a status, indicating whether replication is lagging at that peer. The
number of database transaction log files held by replication is output, followed by the
MySQL log sequence number (that is, LSN) of the earliest transaction held by the MySQL
replication daemon. The LSN is of the format filenum/transactionNum.
 The filenum corresponds to the transaction log file number for transaction logs in
/usr/local/broadworks/bw_base/persistent.
 The transactionNum is an internal transaction number.
11.3 Install Cluster of Servers

In a redundant installation, the installation process prompts the operator for basic cluster
information:
 Number of peers
 For each peer:
− Primary or not – The concept of primary “installation server” applies to the
Application Server, Element Management System, Network Server, and
Messaging Server. There is always one and only one primary server in a cluster.
The concept of a primary server for Call Processing applies only to the
Application Server.
− Hostname – The canonical host name of the server.
− Address (signaling address) – A routable address used by the data replication
processes. It can be hostname, Fully Qualified Domain Name (FQDN), or IP
address. Hostname is recommended.
The information in the following installation example is gathered on the primary server.
…
Will this server be a member of a redundant cluster? (y/n) [n]?y
Is this server the primary peer (the first one installed) (y/n) [y]?
This server is the primary peer. Its installation is a bit more involved
and requires you to enter the list of all peers in the cluster. These
other peers (secondaries) will use this information to configure
themselves.
You must enter a list of hostname/address pairs identifying each peer.
The hostnames should be the servers’ canonical hostname

(to know the canonical hostname of a server, type ‘hostname’
at the command prompt). The hostname for the primary server
is retrieved automatically.
The addresses must be routable and will be used when configuring

replication. They can be hostnames, decimal dotted IP addresses,
or fully qualified domain names.
Note that it is possible to add a server to an existing cluster

without re-installing the existing servers. Please consult the
documentation.
What is the ADDRESS of THIS server? [MTLNS04]

©
Redundant server settings:
Current server hostname is MTLNS04.
Primary server hostname is MTLNS04.
Currently configured peer(s) (list of hostname/address):
MTLNS04/MTLNS04 (primary)
Replication Port: 17888
Enter an additional peer hostname (q to quit, c to clear the list):

MTLNS01
Enter peer address: MTLNS01
Enter an additional peer hostname (q to quit, c to clear the list): q
Please provide a TimesTen replication port [17888]:
Do you accept these settings (y/n) [n]?y
NOTE: The individual FQDN for this server, mtlns04.mtl.broadsoft.com, was

automatically taken from the /etc/hosts file.
When run on the secondary server, the installation process automatically retrieves the
cluster information from the primary server. The only input required is the routable
address of the primary server.
…
Will this server be a member of a redundant cluster? (y/n) [n]?y
Is this server the primary peer (the first one installed) (y/n) [y]?n
This server is a secondary server. All that is needed to configure

redundancy is the routable address of the primary server.
What is the routable address of the PRIMARY server? [] MTLNS04

…
Secure Shell (SSH) is installed on all Network Servers, Application Servers, and
Messaging Servers to support the need for automated replication control to issue remote
commands from one server to the other in a secure manner. During installation, keys are
automatically exchanged between the peers.
11.3.1 Element Management System Additional Parameters

When installing a redundant Element Management System, the administrator is prompted
for additional parameters:
 Heartbeat Interval – Time interval between heartbeat events on the active server (the
heartbeat event is written into the database and replicated to the backup server
database).
 Failover Interval – Time interval between two readings of the heartbeat event on the
backup server. In case the heartbeat event is not set in the database, the backup
servers increment a counter of consecutive heartbeat failure events.

©
 Retry Count – Number of heartbeat failure events before the backup server initiates a
failover.
Example: In case the failover time is set to 30 seconds, and the retry count is set to
“2”, it takes up to 60 seconds for the backup server, before taking over the failed
active server.
NOTE: The data received by the backup server (SNMP traps) during the failed heartbeat
interval is kept and committed into the database once the backup server takes over the active
server. This implies that there should not be any data lost for as long as the network elements
send their data to both Element Management System servers simultaneously.
Following is a sample output for the installation of an Element Management System.

...
Do you want to configure redundancy? (y/n) [y]?y
Will this server be a member of a redundant cluster? (y/n) [y]?
Is this server the primary peer (the first one installed) (y/n) [y]?
themselves.
The hostnames should be the servers' canonical hostname

(to know the canonical hostname of a server, type 'hostname'


documentation.
What is the ADDRESS of THIS server? [mtlems01]

Data redundancy server settings:
Current server hostname is mtlems01.
Primary server hostname is mtlems01.
mtlems01/mtlems01 (primary)
Active server heart beat interval: 20.
Standby server fail over interval: 30.
Standby server retry counts: 2.

mtlems02
Enter peer address: mtlems02
What is the heart beat interval (in seconds)

when running as the active server? [20]

©
Note: the fail over interval must be
longer than the heartbeat interval.
What is the fail over interval (in seconds)
when running as the standby server? [30]
What is the number of retries before activating

the fail over when running as the standby server? [2]
11.4 Network Server and Application Server Synchronization

Application Server group and user add/delete transactions are automatically pushed to the
Network Server. It is possible that the Application Server and Network Server are out of
synchronization, for example, if auto-synchronization is disabled or it is not functioning.
The Application Server and Network Server can be re-synchronized by manually dumping
the Application Server users and groups, transferring the file to the Network Server, and
loading the file on the Network Server.
The Network Server overlays the information on top of existing information.
To re-synchronize the Application Server and Network Server:
1) Create a temporary directory and set full permissions. Example:
$ mkdir /export/home/bwadmin/asdumpOutput
$ chmod 777 /export/home/bwadmin/asdumpOutput
NOTE: The CLI runs as the bworks user, which cannot write to all directories on the server.
Creating and using a temporary directory to store the ASDump information is then
recommended.
2) From the Application Server CLI System/Util/ASDump level, type:

dump /export/home/bwadmin/asdumpOutput/asdump.txt
3) Transfer the asdump.txt file to the Network Server bwadmin home directory. The file
should be transferred as text.
4) From the Network Server CLI System/Util/ASUpload level, type:
upload <hostingNeName> /export/home/bwadmin/asdump.txt
/var/broadworks/log.txt
… where <hostingNeName> is the name of the HostingNe representing the

Application Server on which the ASDump was performed.
11.5 Add New Peer to Redundant System
11.5.1 Application Server, Network Server, and Messaging Server

To add an additional peer to an already redundant system, the BroadWorks Application
Server, Network Server, and Messaging Server must adhere to the following procedure. If
more than one additional peer is to be added, perform the complete procedure for each
new peer at a time.
NOTE 1: Make sure all servers are of the same release.

©
NOTE 2: Make sure all servers are at the same patch level.
1) Install the new server following the instructions for the installation of a stand-alone
server provided by BroadSoft in the BroadWorks Software Management Guide.
NOTE 1: Make sure that all servers from the cluster and the newly installed server have SSH
keys configured properly. For more information, see section 11.9.2 Configure SSH Between
BroadWorks Servers.
NOTE 2: BroadWorks must not be running on the newly installed server (run stopbw).
2) On the primary server, run peerctl add <hostname> <ipaddress> where

<hostname> is the stand-alone server host name.
NOTE: This procedure can also be used to configure two stand-alone servers as one single
cluster. Take into account that one of the stand-alone servers is referenced in this guide as the
cluster server while the other is designated as the new stand-alone node.
3) Verify that the new stand-alone node has been added correctly with peerctl ls.
4) On the new stand-alone node, run peerctl add <hostname> <ipaddress> where
<hostname> is the primary server host name.
5) On the new stand-alone node, run peerctl setPrimary <hostname> where
<hostname> is the primary server host name.
6) Verify that the primary site is properly identified on the new stand-alone with peerctl
ls.
7) Restart the replication on all existing members of the cluster, by running the following
commands on one cluster member at a time.
(a) Stop BroadWorks with stopbw --force
(b) Restart replication with repctl restart.
(c) Start BroadWorks with startbw.
8) On the new stand-alone node, import the database by running importdb.pl
<LocalDSN> <RemotePeer> <RemoteDSN> and press Enter when prompted.
9) Restart the replication on the newly added node with command repctl restart.
10) For the Network Server, update the Network Server routing alias list by adding the
newly added peer.
11) Start BroadWorks on the newly added node with command startbw.
Following is a sample of this procedure.

©
NOTE: In the following example, vm81 (primary) and vm05 are in the same cluster whereas,
vm04 is the stand-alone server to be added to the cluster. The host name used is related to
what the host name command returns locally for each server.
bwadmin@vm81$ peerctl add vm04 vm04

Restart replication on all servers of the cluster using repctl restart.
bwadmin@vm81$
bwadmin@vm81$ peerctl ls
--------------------------------------------------------------------------------
vm81/vm81 primary,unlocked
vm05/vm05 unlocked
vm04/vm04 unlocked
bwadmin@vm81$
bwadmin@vm04$ peerctl add vm81 vm81

Restart replication on all servers of the cluster using repctl restart.
bwadmin@vm04$ peerctl setPrimary vm81
bwadmin@vm04$ peerctl ls
--------------------------------------------------------------------------------
vm04/vm04 unlocked
vm81/vm81 primary,unlocked
bwadmin@vm04$
bwadmin@vm81$ stopbw --force

BroadWorks control script version 22.0
Stopping BroadWorks...
Broadcast message from bworks@vm81 (Mon Mar 6 14:58:03 2017):
===== BROADWORKS CONTROL --- STOP INITIATED ON VM81 =====

Stopping tomcat...
Stopping nsExecution...
Stopping nsProvisioning...
Waiting for core processes to terminate.....
... Done.
Stopping remotexla...
Waiting for core processes to terminate......
... Done.

BroadWorks License Manager process monitor (pid=27841)
BroadWorks License Manager (pid=27871)
bwadmin@vm81$ repctl restart

Stopping file replication (this may take up to 60 seconds, please be patient)
vm05: Checking replication status... Started.
vm05: Checking version... Versions match (NS version Rel_22.0_1.1123)
vm05: Checking patch level... synchronized.

©
vm04: Checking replication status... Stopped.
=> Replication is clear for start up
bwadmin@vm81$ startbw
Starting BroadWorks...
===== BROADWORKS CONTROL --- START INITIATED ON VM81 =====

Starting remotexla...
Starting nsExecution...
Starting nsProvisioning...
Starting tomcat...
bwadmin@vm81$
bwadmin@vm05$ stopbw --force

===== BROADWORKS CONTROL --- STOP INITIATED ON VM05 =====

Stopping tomcat...
... Done.
... Done.

BroadWorks License Manager process monitor (pid=10300)
BroadWorks License Manager (pid=10329)

vm04: Checking replication status... Stopped.
bwadmin@vm05$
bwadmin@vm04$ importdb.pl

©
WARNING: Using the following default values:
-> DSN: NetworkServer
-> Remote DSN: NetworkServer
-> Remote hostname: vm81
Press Ctrl-C now to abort or <Enter> to continue...

Continuing.

Backing up dsn NetworkServer to
/var/broadworks/backup/NS_Rel_22.0_1.1123.170306152900.backup... [done]
WARNING: ALL information in data store NetworkServer will be lost.

Press Ctrl-C now to abort or <Enter> to continue...
Continuing.

Retrieving localhost in remote replication schema ... [vm04]
Performing checkpoint on remote database... [done]
Unloading database from memory... [done]
Destroying database NetworkServer... [done]
Importing the remote database... [done]
Loading database in memory... [done]
Updating statistics... [done]
bwadmin@vm04$

bwadmin@vm04$
The system is now redundant. You can re-use this procedure to add any number of
additional stand-alone servers to a cluster.

©
NOTE: Additionnal system configuration steps may be required after an additional member is
added to the cluster. For instance, DNS or namdefs and /etc/hosts may need to be updated. The
Application Server configuration may need updates in case of Network Server addition.
11.5.2 Element Management System

To add an additional peer to a non-redundant system, use the BroadWorks Element
Management System setup-redundancy script. The setup-redundancy script queries for
the list of new peers, it adds them to BroadWorks, it configures SSH, and then it begins
replication on all original servers so that the new node is ready to import the DSN. All
functionality is built into the script, which prompts the user for the necessary information.
NOTE: Make sure that all servers are of the same release.
1) Following the instructions for the installation of a secondary server provided by

BroadSoft in the BroadWorks Software Management Guide, install the new peer.
Make sure to properly set the primary server name while configuring redundancy.
NOTE: Do not log in to the new peer as bwadmin until step 3, which follows.
2) On the primary server, run setup-redundancy and add a new peer when
prompted. The following is a sample of this procedure.
bwadmin@shoule [/usr/local/broadworks/bw_base/bin]$ ./setup-redundancy
Checking current redundancy set-up
You are about to set-up ems redundancy for Rel_15.0_1.200

Do you wish to continue? (y/n) [y]? y
Setting up ems redundancy for Rel_15.0_1.200
... Loading the configuration properties for the ElementManagementSystem
--- BroadWorks Installation Data Entry ---

Current server hostname is shoule.
Primary server hostname is shoule.
shoule/shoule (primary)
Do you want to configure redundancy? (y/n) [y]?y
Will this server be a member of a redundant cluster? (y/n) [y]?y
Is this server the primary peer (the first one installed) (y/n) [y]?y
themselves.

©
The hostnames should be the servers' canonical hostname
(to know the canonical hostname of a server, type 'hostname'


documentation.
What is the ADDRESS of THIS server? [shoule]


tier3mtl
Enter peer address: 192.168.8.143
What is the heart beat interval (in seconds)

when running as the active server? [20]
Note: the fail over interval must be

longer than the heartbeat interval.
What is the fail over interval (in seconds)
when running as the standby server? [30]
What is the number of retries before activating

the fail over when running as the standby server? [2]
===================================================
Basic settings:

tier3mtl/192.168.8.143
===================================================
Type Ctrl-c to abort installation
Re-spawning as root...
INFO: configuring redundancy from a BroadWorks upgrade to
EMS_Rel_15.0_1.200

©
INFO: running on primary
INFO: Retrieving data from /var/broadworks/installation.conf
INFO: updating my.cnf configuration file
Killing mysqld with pid 12402
Wait for mysqld to exit. STOPPING server from pid file
/usr/local/mysql/mysql_base/var/shoule.pid
081128 17:16:22 mysqld ended
done
nohup: redirecting stderr to stdout
Starting mysqld daemon with databases from
/usr/local/mysql/mysql_base/var
INFO: PRIMARY_HOSTNAME is shoule

INFO: PRIMARY_ADDRESS is shoule
INFO: SECONDARY_HOSTNAME is tier3mtl
INFO: SECONDARY_ADDRESS is 192.168.8.143
INFO: using config-ssh for bwadmin to 192.168.8.143

INFO: Enforcing password for sql user on shoule
INFO: Backing up initial database
SUDO_USER is bwadmin
Backing up ems db...
Backing up from directory /usr/local/mysql/mysql_base/var/WebNmsDB

Backing up to /var/broadworks/db
INFO: preserving old backup
Ems back up completed
INFO: Generating sql privilege statements for peer 192.168.8.143

INFO: Running sql redundancy script on shoule
INFO: Generating sed files for configuration changes on shoule
INFO: generating redundancy parameter data for bw auto config
INFO: adding peer (tier3mtl) to /etc/hosts
INFO: configuring
/usr/local/broadworks/EMS_Rel_15_0_1.200/public_html/conf/database_param
s.conf
INFO: configuring
/usr/local/broadworks/EMS_Rel_15.0_1.200/public_html/conf/FailOver.xml
INFO:
/usr/local/broadworks/EMS_Rel_15.0_1.200/public_html/conf/serverparamete
rs.conf already configured
INFO: configuring
/usr/local/broadworks/EMS_Rel_15.0_1.200/conf/setEnvFE.sh
bwadmin@192.168.8.143's password:
bwadmin@192.168.8.143's password:
Done.
3) Log in as bwadmin and perform the post installation on the new peer.
11.5.3 Profile Server

To add an additional peer to a system (whether or not redundant), the BroadWorks Profile
Server uses the config-redundancy script. The config-redundancy script requests the list
of new peers, adds them to BroadWorks, configures SSH, and begins file replication. All
functionality is built into the script, which prompts the user for necessary information.

©
NOTE: Make sure all servers are of the same release.
1) Following the instructions for the installation of an additional server provided by

BroadSoft in the BroadWorks Software Management Guide, install the new peer.
Make sure to properly set the primary server name while configuring redundancy.
2) On all servers, run config-redundancy and add new peers when prompted. Following
is a sample of this procedure.
bwadmin@PS01$ ./config-redundancy
-> Data gathering <-

--------------------------
Non-Redundant system.
--------------------------

PS02
Enter peer address: PS02
-> Confirmation <-
This is the New redundancy configuration:

--------------------------
PS02/PS02
--------------------------
Do you want to keep the following configuration? (y/n) [y]?y
-> Adding new peer <-

Adding PS02/PS02...
-> Configure Secure Shell <-
----------------------------------------------
---- SSH CONFIGURATION TOOL version 2.2.15 ----
-> Setting default settings <-
Setting 'StrictHostKeyChecking no'
-> DNS Sanity test <-
[###############]
[...............]
-> Peer reachability test <-
[###]
[...]
-> Creating SSH keys <-
Creating keys for bwadmin@PS02...
Creating keys for bwadmin@PS01...
-> Keying SSH <-

Preparing bwadmin@PS01 for keying...
Sharing keys with bwadmin@PS02...

Cleaning public keys for bwadmin@PS02...

©
Pushing local public keys...
Pulling remote public keys...
Sharing keys with bwadmin@PS02... [done]
-> Fully meshing SSH peers <-
-> Recursing with bwadmin@PS02 <-

Pushing config-ssh script to bwadmin@PS02...
Launching config-ssh on bwadmin@PS02...
-> Setting default settings <-

Setting 'StrictHostKeyChecking no'
[###############]
[...............]
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@MTLSOL-14.mtl.broadsoft.com for keying...
Sharing keys with bwadmin@PS01...

Cleaning public keys for bwadmin@PS01...
Pushing local public keys...
Pulling remote public keys...
Sharing keys with bwadmin@PS01... [done]
-> Testing ssh configuration <-

Testing bwadmin@PS02... [done]
---- SSH CONFIGURATION TOOL completed ----
---- config-redundancy [done] ----
3) On the new server, import the file repository files from the existing peer. As the
bwadmin user, run the following command on the new server.
$ rsync -vazurSHl <peerServer>:<file repository root>/. <file repository
root>/.
11.6 Remove Peer from Redundant System

This procedure applies to the Network Server and Messaging Server.
NOTE: For the Application Server, there are only two nodes; removing one leaves you with a
non-redundant configuration.
To remove a peer from a redundant system, type the following commands:

 peerctl del
 repctl restart – on each node in the cluster

©
11.7 Recover Faulty Redundant Installation
If the primary server is installed and activated before installing the redundant server, the
SSH configuration is incorrect. Such a faulty installation may have three symptoms
depending on the base system:
 SSH is started and bwadmin does not exist.
 SSH is started and bwadmin exists, but SSH is not configured.
In both of these cases, the following message appears on the screen after starting
replication.
The authenticity of host ‘redsrv1 (192.168.8.63)’ cannot be established.
RSA key fingerprint is 96:1a:9a:a7:cb:6e:95:26:68:70:cc:6e:39:b0:90:9f.
Are you sure, you want to continue connecting (yes/no)? Please type ‘yes’
or ‘no’.
 SSH is not started (installed or not).

In this case, the following message appears.
Secure connection to redsrv2 refused. unexpected EOF in read_timeout
To correct the situation, the following commands should be entered at the prompt on the
primary server.
$ repctl stop
$ config-ssh –createKeys <redsrv1> <redsrv2> …
$ repctl start
11.8 Post-installation Control

Peer information can be accessed via the server CLI. An operator can also lock the local
database or the database of a peer through the CLI to prevent modifications during a
maintenance window.
NS_CLI/System/Peering> ?
0) get : show replication parameters-related attributes
1) set : modify replication parameters-related attributes
2) lock : to lock the database
3) start : to start system replication
4) status : to obtain status of system replication
5) stop : to stop system replication
6) unlock : to unlock the database
7) Peers : go to level Peers

(repeat)
NS_CLI/System/Peering> peer
NS_CLI/System/Peering/Peers> get
HostName Address State
=========================================================================
mtl64lin17.mtl.broadsoft.com mtl64lin17.mtl.broadsoft.com unlocked
mtl64lin10.mtl.broadsoft.com mtl64lin10.mtl.broadsoft.com unlocked
2 entries found.

©
11.9 SSH/RSYNC Configuration
SSH is used as the remote shell invocation method as well as for secure file
synchronization. To ensure trouble-free operation, the following steps should be taken.
11.9.1 Create SSH Keys (Completed Automatically During Installation)

 The SSH keys are created automatically during installation. A passphrase can be
used or not used. The recommended solution is not to use any pass phrase and to
rely on the overall network security, to ensure a proper usage of public keys.
However, the safest solution is to set up a complex pass phrase that must be used to
access the server. Keys can be regenerated at any time using the following three
commands.
$ /usr/local/bin/ssh-keygen –t rsa1 –f /export/home/bwadmin/.ssh/identity
$ /usr/local/bin/ssh-keygen –t dsa –f /export/home/bwadmin/.ssh/id_dsa
$ /usr/local/bin/ssh-keygen –t rsa –f /export/home/bwadmin/.ssh/id_rsa
 For more information on how to use a pass phrase, see the OPENSSH
documentation available from www.openssh.org.
11.9.2 Configure SSH Between BroadWorks Servers
11.9.2.1 Automated Configuration

SSH keys can be shared automatically, and the known_host list can be filled between
cluster members with the config-ssh utility.
ws1.broadsoft.com$ config-ssh –h
Usage:/usr/local/broadworks/bw_base/bin/config-ssh [-createKeys] peer1
user@remotehost...
SSH configuration utility is used to create ssh keys for every peer
within a cluster. The utility then propagates the ssh keys to all peers
to facilitate access between each other. Any UNIX system that has
ksh, ssh and rsync installed can use this utility script.
Option
------
-createKeys is used to create ssh keys for every peer
ws1.broadsoft.com$ config-ssh as1.broadsoft.com

----------------------------------------------
---- SSH CONFIGURATION TOOL version 1.14 ----
[###############]
[...............]
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@ws1.broadsoft.com for keying...
Sharing keys with bwadmin@ws1.broadsoft.com...

Pulling remote keys from bwadmin@ws1.broadsoft.com...
Adding bwadmin@ws1.broadsoft.com new keys to
bwadmin@ws1.broadsoft.com authorised_keys...
Sharing keys with bwadmin@ws1.broadsoft.com... [done]
Sharing keys with bwadmin@as1.broadsoft.com...

Pulling remote keys from bwadmin@as1.broadsoft.com...

©
Adding bwadmin@as1.broadsoft.com new keys to
Pushing public keys unto bwadmin@as1.broadsoft.com...
bwadmin@as1.broadsoft.com authorised_keys...
Sharing keys with bwadmin@as1.broadsoft.com... [done]
-> Fully meshing SSH peers <-
-> Recursing with bwadmin@solns <-

Pushing config-ssh script to bwadmin@ as1.broadsoft.com...
Launching config-ssh on bwadmin@ as1.broadsoft.com...
[###############]
[...............]
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@as1.broadsoft.com for keying...
Sharing keys with bwadmin@as1.broadsoft.com...

Pulling remote keys from bwadmin@as1.broadsoft.com...
Sharing keys with bwadmin@as1.broadsoft.com... [done]
Sharing keys with bwadmin@ws1.broadsoft.com...

Pulling remote keys from bwadmin@ws1.broadsoft.com...
Pushing public keys unto bwadmin@ws1.broadsoft.com...
Sharing keys with bwadmin@ws1.broadsoft.com... [done]
---- SSH CONFIGURATION TOOL completed ----
11.9.2.2 Manual Configuration
11.9.2.2.1 Share SSH Keys

To access the server without a pass phrase in SSH, public keys must be in
~/.ssh/authorized_keys or ~/.ssh/authorized_keys2 of the remote server.
To locate the public keys in the proper directory:
#On each servers participating in a cluster (including the current
hosts):
$ cat $HOME/.ssh/identity.pub > $HOME/.ssh/authorized_keys.`hostname`
$ cat $HOME/.ssh/id_rsa.pub > $HOME/.ssh/authorized_keys2.`hostname`
$ cat $HOME/.ssh/id_dsa.pub >> $HOME/.ssh/authorized_keys2.`hostname`
#for each other server (than the current)

$ ftp $SERVER
cd .ssh
bin
put authorized_keys.`hostname` (replace `hostname` with the real value)
put authorized_keys2.`hostname`(replace `hostname` with the real value)
#then, on each server

©
$ cat $HOME/.ssh/authorized_keys.* >> $HOME/.ssh/authorized_keys
$ cat $HOME/.ssh/authorized_keys2.* >> $HOME/.ssh/authorized_keys2
NOTE: Pass phrases are like passwords and should be changed on a periodic basis.
Whenever pass phrases change, public identities need to be redistributed for SSH to continue
working.
11.9.2.2.2 Known_hosts List

Fill the known_hosts list by connecting to each redundant server using SSH, which also
includes the current hosts. SSH then asks if you want to add the new server to the list of
known hosts.
AS1$ ssh bwadmin@192.168.12.7
The authenticity of host ‘192.168.12.7 (192.168.12.7)’ cannot be
established.
RSA key fingerprint is 76:9a:9e:06:12:06:4a:84:cc:fc:33:8b:c3:db:24:87.
Are you sure you want to continue connecting (yes/no)?yes
Warning: Permanently added ‘192.168.12.7’ (RSA) to the list of known
hosts.
AS1$
11.9.3 Use SSH

To access another server in the cluster, by issuing ssh bwadmin@serverName you get
access to the server’s serverName.
Example:
$ ssh bwadmin@MTLAS-4
Last login: Wed Feb 6 10:32:25 2002 from denali
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
###### # #
# # ##### #### ## ##### # # # #### ##### # # ####
# # # # # # # # # # # # # # # # # # # #
###### # # # # # # # # # # # # # # # #### ####
# # ##### # # ###### # # # # # # # ##### # # #
# # # # # # # # # # # # # # # # # # # # #
###### # # #### # # ##### ## ## #### # # # # ####
BroadWorks and its documentation are protected by copyright law and international
treaties. Unauthorized reproduction or distribution of this software, or any part
thereof, may result in severe civil and criminal penalties, and will be prosecuted
to the maximum extent possible under the law.
MTLAS-4$
If you issue the SSH bwadmin@serverName command, the command is run on the
server serverName.
Example:
$ ssh bwadmin@as4 /usr/local/broadworks/bw_base/bin/repctl status
File Replication pid =
Datastore name = ApplicationServer
Replication Agent Policy = manual
Replication Manually Started = false

©
12 Disaster Recovery Strategy
Disaster recovery, defined as the ability to return servers and applications to their working
states following unexpected physical damage or system failures causing data loss, is an
issue that encompasses the high-level design and maintenance of entire back-end
infrastructures. As such, it is outside the direct scope of BroadWorks maintenance and
best considered in close collaboration with your IT department (and in light of best
practices within the industry). However, certain general approaches and guidelines can
be considered with respect to BroadWorks servers.
Disaster recovery from the BroadWorks perspective entails two stages, first creating a
replacement server platform with matching operating system (OS) and BroadWorks
software level, and then recreating the user information and configuration data of the
BroadWorks component and reintegrating the server into the BroadWorks network.
Replacement servers must have the same settings as the servers they replace,
incorporating all the following elements:
 OS image
 OS patches
 UNIX/LINUX host name
 IP address(es) and configuration
 /etc/hosts file
 /etc/hostname.* files
 DNS configuration
 NTP configuration
 Security configuration (for example, JASS)
 BroadWorks release
 BroadWorks patch level
While the platform type of the server is not required to remain the same, it is
recommended that the new server have the same hardware footprint (CPUs, RAM).
Application Servers, Network Servers, and Messaging Servers do require that the new
server have the same OS type as the old server (Solaris or Linux).
If servers have multiple IP interfaces, then the replacement servers must also have the
same interfaces available and they must be defined.
12.1 Recover Server OS

Selecting a method for server restoration depends on a number of factors, such as
available capital investments for backup servers and organizational standards for
acceptable recovery times. The following is an overview of common approaches that you
can use successfully with BroadWorks.

©
12.1.1 Hot Standby Servers
In this scenario, a hot standby server is available for each BroadWorks server type. The
hot standby servers are running the same BroadWorks releases and are at the same
BroadWorks patch levels as their live counterparts (that is, the standby servers are
patched whenever the live servers are patched). Generally, the Network Server,
Application Server, and the Messaging Server standbys are installed as non-redundant
servers, and have blank databases. This helps to speed up reintegration into the
BroadWorks production network.
When an existing server must be replaced, the administrator reconfigures the standby as
follows:
 Configures the IP address to match the server to be replaced.
 Configures the host name to match the server to be replaced.
 Sets the local peer information to match the host name.
For example, if the standby was previously configured as “backup”, then the peerctl ls
command returns “backup” as local peer name.
bwadmin@ne1$ peerctl ls
-----------------------------------------------------------------------
backup/backup primary,unlocked
In the preceding example, the following commands need to be run on the

replacement server to set the peers properly.
bwadmin@ne1$ peerctl set backup ne1 ne1
 Moves the standby to the same subnet as the server to be replaced.

 Powers up the server and connects it to the switch.
Once these tasks are completed, the administrator can recreate the BroadWorks
application data and reintegrate the server back into the BroadWorks production network
as described in section 12.2.3 BroadWorks Server Restoration Procedure.
12.1.2 On-Demand Rebuild

As an alternative to maintaining standby copies of all BroadWorks servers, replacement
servers can be rebuilt on demand with the OS and BroadWorks components required to
emulate the failed server.
OS reinstallation can be done manually. However, for speed and consistency it is
recommended that you configure a Solaris JumpStart or Linux KickStart server, ensuring
that the replacement server is built to the same specification as the live servers (that is, the
same OS patch level). This approach also ensures that any customer-specific packages
are loaded at the same time.
Some providers opt to have a single OS pre-built standby server, which can be used to
replace any server in the BroadWorks network. Ideally, this single backup server platform
has a hardware footprint equal to the "largest" BroadWorks server. For example, in a
BroadWorks deployment with a V440 for the Application Server, a V210 for the Network
Server, and a V120 for the Media Server, the standby replacement server would be a
V440 to support the same capacity should the Application Server need to be replaced.
This single OS pre-built standby server saves the three to four hours that it can take to
reinstall and reconfigure the operating system on the replacement server.

©
Some providers opt to flash archive each production server, so that the JumpStart or
KickStart process automatically restores the OS with the exact same configuration as the
existing server.
Regardless of how the replacement server is built, it needs to be configured at the OS
level with the same information as the server to be replaced. For example,
 Hostname
 IP address
 /etc/hosts file
 /etc/hostname.* files
 Default router
 DNS configurations
 NTP configurations
Once the server is rebuilt with the required OS level and configuration, the corresponding
BroadWorks application can be installed on the replacement server, and any required
BroadWorks patches can be applied to bring the replacement to the same software
release as the production server.
For Application Servers, Network Servers, and Messaging Server s, it is typically easier to
install the component as a non-redundant server. In general, however, the configuration
information entered as part of the fresh install on the replacement server is not important,
as it is overwritten or reconfigured when the server is reintegrated into the live network.
Fresh installation and patching of a BroadWorks component should not take more than 30
to 60 minutes.
For providers that have opted to flash archive each production server, the flash archive
automatically restores the BroadWorks application level that was present at the time the
archive was made. Complications can arise if the version of the BroadWorks component
in the archive does not match the version of the BroadWorks component to be replaced,
as an upgrade to the production version would be required. It is therefore recommended
that the server be flash archived at least after every major BroadWorks release upgrade.
This requires only incremental BroadWorks patch applications to get the replacement
server to the same state as the production server.
Once the BroadWorks application on the replacement server is at the same level as that
on the production server, the administrator can recreate the BroadWorks application data
and reintegrate the server back into the BroadWorks production network as described in
section 12.2.3 BroadWorks Server Restoration Procedure.
12.1.3 Incremental Backups

In this scenario, the provider uses a third-party backup utility to periodically back up all
BroadWorks servers in the production network. There are a number of different
approaches within this scenario depending on the specific backup application used, but
the result from a BroadWorks perspective is the same in all cases. The replacement
server rebuilt from the full and incremental backups must end up with the same software
level as the server it replaces, both for the OS and for the BroadWorks component.
After restoring a server in this scenario, administrators still have to reintegrate the
replacement server into the BroadWorks network following the procedure outlined in
section 12.2.3 BroadWorks Server Restoration Procedure. However, not all the steps
apply in all cases. If the rebuilt server is actually more recent than the last BroadWorks
system backup, the instructions for restoring the BroadWorks system backup do not need
to be carried out.

©
12.2 BroadWorks Server Restoration
As stated previously, replacing a BroadWorks server requires that the replacement server
be at the same OS and BroadWorks level as the server to be replaced. Once that is
accomplished, the replacement server can be repopulated with user and application data if
required, and reintegrated into the production network.
The reintegration procedure consists of a number of manual configuration steps, and uses
the contents of the BroadWorks system backup to configure the replacement server with
the most recent information backed up from the server to be replaced. If the server to be
replaced has a live peer in the network with up-to-date information (for example, database,
user file synchronization), then it is obtained from the peer instead of the backup file. This
applies only to the Application Server, Network Server, and Messaging Server.
The amount of time required to fully restore a BroadWorks server is the sum of the time
required to:
1) Create a replacement server at the proper OS and BroadWorks level.
2) Mount and power the server in the proper location.
3) Run the BroadWorks restoration procedure described (as follows):
The first two variables are outside the scope of BroadWorks maintenance, and depend on
the OS restoration scheme in use. The time required for the BroadWorks Server
Restoration procedure depends on both the server component and the size of the
customer base. For example, a Media Server restoration takes only minutes, while an
Application Server restoration is bound to the time it takes to import data from the
secondary server, which can take as long as one hour on a larger cluster. In general, the
BroadWorks Server Restoration procedure should not take more than two hours on any
server type for any customer base or database size.
NOTE: On the BroadWorks Database Server, the restoration of the centralized database can be
significantly longer, depending on size of the database and the speed of the I/O subsystem. It is
recommended to evaluate the restoration time in a lab environment to determine realistic time
requirements.
12.2.1 BroadWorks System Backup

The system backup created by the bwAutoBackup.sh script (run manually or as a
scheduled task) only contains BroadWorks items that change over time, such as:
 Directories that contain server-specific configuration files and user files (for example,
greetings)
 A backup of the database, if applicable for that server type
For a list of what is backed up on each server, see the
/usr/local/broadworks/bw_base/conf/bwSystemBackup.conf file.
The items are tarred up, then gzipped into a single archive file and placed in the directory
specified in bwSystemBackup.conf. By default, this directory is
/usr/local/broadworks/bw_base/persistent/backups.

©
When you unzip and untar the backup file in the / directory, all included directories and
files present on the local host are overwritten with the content of the backup, and a copy of
the backed-up database is placed in the default backup directory. BroadWorks
recommends scheduling the BroadWorks system backup to run on a daily basis. These
backups must be periodically transferred to another location to make sure that in the event
of a server loss, the backups are available.
NOTE: Starting with Release 18.0, Web Server (HTTP) certificates and keys are now backed up
by the system backup and they do not have to be manually backed up when they are changed
by the service provider.
12.2.2 Licensing
To start the BroadWorks application on the replacement server, you require license files
that contain the hostid of the replacement server. To obtain new license files, contact
BroadSoft. If you use pre-configured recovery servers (either hot-standby, or pre-
configured OS standby, described above), you may be able to get the host IDs of the
standby servers added to existing license files, to avoid having to change license files in
the event of a swap. For more information, contact your BroadSoft Project Manager.
12.2.3 BroadWorks Server Restoration Procedure

Use the following procedures to replace existing BroadWorks servers. These procedures
assume that the replacement server has been built out to the same BroadWorks release
and patch level as the server it is replacing.
12.2.3.1 Media Server Replacement

1) Stop the BroadWorks application and the Configuration Agent on the replacement
server.
bwadmin@host$ stopbw
bwadmin@host$ configdctl stop
2) Copy the last BroadWorks system backup created from the server to be replaced, to
any directory on the replacement server. As the bwadmin user, run the following
commands.
bwadmin@host$ gunzip <backupfile_name.gz>
bwadmin@host$ tar xvf <backupfile_name>
This restores the last backup of the production server's configuration information to
the replacement server. Because the backup .tar file stores the relative paths of all
included files, you need to extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you run the tar xvf command.
Note that the restore operation should be executed as root when performed from the
system root directory (/).
3) Make sure that valid license.txt and license.sig files are in place in the
/usr/local/broadworks/bw_base/conf directory.

©
4) Start the Configuration Agent.
bwadmin@host$ configdctl start
5) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.
6) Start BroadWorks.
bwadmin@host$ startbw
12.2.3.2 Application Server Replacement

Throughout this example, the new replacement server is referred to as “ne1”, while the
existing peer is called “ne2”.
1) This step is not necessary if the new replacement server is already ready. If the
server to be recovered is down for more than a few hours, set the remaining peer as
the primary, if not already done, and stop replication.
bwadmin@ne2$ peerctl setPrimary ne2
bwadmin@ne2$ peercmd repctl stop
2) If required, install BroadWorks on the replacement server and patch it to the level of
the active peer.
3) Stop the BroadWorks application, replication, the SNMP agent, the License Manager,
and the Configuration Agent on the replacement server if they are running.
bwadmin@ne1$ stopbw
bwadmin@ne1$ repctl stop
bwadmin@ne1$ snmpdctl stop
bwadmin@ne1$ lmdctl stop
bwadmin@ne1$ configdctl stop
4) Re-mesh the SSH keys. On all peers, clean up the .ssh directory by running the
following.
bwadmin@ne1$ rm -Rf /export/home/bwadmin/.ssh
5) As bwadmin, from either server, run the following.

bwadmin@ne1$ config-ssh -createKeys <ne1> <ne2> <other_peer_server>
<other_peer_server> can be the host name of the peer if the /etc/hosts file on the
current host is configured properly with the peer host name.
6) Reconfigure database replication peers.
If any peer server is still in service, as bwadmin, run the peerctl ls command on the
existing peer server host to get a list of peers. Following is an example output of
peerctl ls from the existing server.
-----------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,locked

©
7) Given the previous examples, the following commands would be run on the
bwadmin@ne1$ peerctl add ne2 ne2
If the replacement server is the primary, the peerctl add command is only needed to
add the existing peer, which is the secondary.
8) Verify that the information is consistent across all peers, with the exception of the state
on the new peer, which indicates “unlocked” while it shows “locked” on all other peers.
bwadmin@ne1$ peercmd peerctl ls
==== BroadWorks Network Command Spawning Tool version 1.2 ====
=> Executing 'peerctl ls' locally <=

------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,unlocked
=> Spawning 'peerctl ls' to ne2 <=

------------------------------------------------------------------
ne1/ne1 unlocked
9) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. If you want to restore files to the directory in
which they were originally located when the backup was made, perform the restore
from the system root directory (/). As the root user, run the following commands.
root@ne1$ gunzip <backupfile_name.gz>
root@ne1$ tar xvf <backupfile_name>
The localization information backup is stored in a file called internationalization.tar.gz.

To re-import the localization information, an operator must use the internationalize.pl –
import AS-localization.tar.gz command. For more information, see the BroadWorks
Application Server CommPilot Portal Customization and Localization Guide.
10) With reference to files that may have changed since the system backup, some
replication-required file types are automatically resynchronized once the new server is
brought back on line within a short period (less than 30 minutes). These files include
items such as web branding files and Device Management files. For a full list of the
automatically resynchronized directories, see file
/usr/local/broadworks/bw_base/conf/rep.conf. Other user/group-related replication-
required files such as custom voice mail greeting, device configuration, and Auto
Attendant announcements are not synchronized automatically to the new server once
it comes back on line, as they are only synchronized on modification. To make sure
that these files are resynchronized properly, the following commands should be
executed from the source (existing server, “ne2” in this example) to manually trigger
an rsync to the destination (new server, “ne1” in this example), making sure to replace
the <Destination-Server-Hostname> with the appropriate host name.
bwadmin@ne2$ /usr/bin/rsync -vazSHle '/usr/bin/ssh -o batchmode=yes'
/var/broadworks/userfiles/customMediaFiles <Destination-Server-
Hostname>:/var/broadworks/userfiles

©
bwadmin@ne2$ /usr/bin/rsync -vazSHle '/usr/bin/ssh -o batchmode=yes'
/var/broadworks/IpDeviceConfig <Destination-Server-
Hostname>:/var/broadworks/
NOTE: The path to SSH may vary. On Linux, SSH is typically located under /usr/bin/ssh
instead of /usr/local/bin/ssh. If both files are available, /usr/bin/ssh should be used.
11) If the Application Server had been localized, web localization should be automatically
preserved from the system backup; however, any tone or announcement
customization is not part of the system backup and must be recreated. These can be
copied from the existing server. Depending on how localization was performed, these
custom files can be in a number of locations as follows:
− /var/broadworks/AS_Rel<xxxxxx>/sysprompts
− /var/broadworks/userfiles/<country_locale>
− /var/broadworks/userfiles/tones
12) Start the Configuration Agent, the License Manager, and the SNMP agent by running
the following command on the new replacement server.
bwadmin@ne1$ configdctl start
bwadmin@ne1$ lmdctl start
bwadmin@ne1$ snmpdctl start
13) Make sure that valid license.zip file is installed on the server (install-licen se.pl
/path/to/license.zip).
14) If stopped, restart the BroadWorks replication on the remaining peer.
bwadmin@ne2$ repctl restart
15) Import the database from the existing peer server. As the bwadmin user, run the
following command on the replacement server.
bwadmin@ne1$ importdb.pl [appserver <peer_ADDRESS> appserver]
16) Restart the replication on the new replacement server.

bwadmin@ne1$ repctl start
17) As bwadmin, run the following command on the secondary server.

bwadmin@ne1$ synchcheck_basic.pl -a

19) If the server replaced was originally the primary server, set it back as primary, if it has
not been done yet.
20) Verify that the information is consistent across all peers, with every peer unlocked.
21) As bwadmin, run the following command.

©
bwadmin@ne1$ startbw
NOTE: If the former secondary server (ne2) was set as primary during this procedure and was
restarted for maintenance reasons before the end of the procedure, a lock and unlock cycle
(AS_CLI/Maintenance/ManagedObjects>) is required to restart the redundancy link.
12.2.3.3 Network Server and Messaging Server Replacement

Throughout this example, the new replacement server is referred to as “ne1”, while the
existing peer is called “ne2”.
1) On the remaining peer in the cluster, set the local peer as primary (if not already set),
and stop replication. This prevents any attempt to synchronize data to/from the
replacement server during the duration of the recovery procedure.
bwadmin@ne2$ peercmd peerctl lock
bwadmin@ne2$ peercmd repctl stop
2) If required, install BroadWorks on the replacement server, and patch it to the level of
the active peer.
3) Stop the BroadWorks application, replication, and the Configuration Agent on the
replacement server if it is running.
bwadmin@ne1$ stopbw
bwadmin@ne1$ repctl stop
bwadmin@ne1$ configdctl stop
4) Re-mesh the SSH keys. On all peers, cleanup the .ssh directory by running the
following.

bwadmin@ne1$ config-ssh -createKeys <ne1> <ne2> <other_peer_server>
6) Reconfigure database replication peers.
If any peer server is still in service, as bwadmin, run the peerctl ls command on the
existing peer server host to get a list of peers. Following is an example output of
-----------------------------------------------------------------------
ne1/ne1 unlocked
On the replacement server, you must make sure that the peer list is the same as it is
on the in-service peer.

©
In case of a hot standby replacement server, the local peer name in the replication
configuration data must have been altered to the local host name, as indicated in
section 12.1.1 Hot Standby Servers.
7) Given the above examples, the following commands would be run on the replacement
server to set the peers properly.
bwadmin@ne1$ peerctl add ne2
8) Verify that the information is consistent across all peers, with the exception of the state
on the new peer, which indicates “unlocked” while it shows “locked” on all other peers.
==== BroadWorks Network Command Spawning Tool version 1.2 ====
=> Executing 'peerctl ls' locally <=

------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,unlocked
=> Spawning 'peerctl ls' to ne2 <=

------------------------------------------------------------------
ne1/ne1 unlocked
==== BroadWorks Network Command Spawning Tool [done] ====
any directory on the replacement server. If you want to restore files to the directory in
which they were originally located when the backup was made, perform the restore
from the system root directory (/). As the root user, run the following commands.
root@ne1$ gunzip <backupfile_name.gz>
root@ne1$ tar xvf <backupfile_name>
10) On the Network Server, the only files that are replicated across peer servers are the
web branding files. Any web-branding file that was changed on the existing/running
Network Servers since the system backup is automatically resynchronized to the new
server within a short period (less than 30 minutes).
11) Start the Configuration Agent by running the following command.
bwadmin@ne1$ configdctl start
13) Restart the BroadWorks replication on the already active peer(s).
bwadmin@ne2$ repctl restart
14) Import the database from the existing peer server. As the bwadmin user, run the
following command on the replacement server.
bwadmin@ne1$ importdb.pl [networkserver <peer_ADDRESS> networkserver]
If the peer server is not in service, restore the database that was included in the last
BroadWorks system backup archive.

©
The parameters to importdb.pl are optional; if importdb.pl is called without any
arguments, it guesses the parameters and prompts for confirmation.
15) As bwadmin, run the following commands.
bwadmin@ne1$ repctl start
bwadmin@ne1$ synchcheck_basic.pl -a
Note that repctl start also starts file replication, which makes sure that any user files
that have changed since the last BroadWorks system backup (for example, greetings)
are replicated to the replacement server within the first 10 minutes of operation.
The syncheck_basic.pl is used to detect discrepancies between the databases on the
peer. If an error is reported, follow the recommendation to re-sync the database. Any
attempt to use the replacement peer in a redundant configuration depends on
properly synchronized databases.
16) Unlock the BroadWorks database on every peer.
bwadmin@ne2$ peercmd peerctl unlock

18) As bwadmin, run the following command.
bwadmin@ne1$ startbw
19) If the server replaced was originally the primary server, set it back as primary.
20) Verify that the information is consistent across all peers, with every peer unlocked.
12.2.3.4 Xtended Services Platform Replacement

server.
commands.
This restores the last backup of the production server’s configuration information to
included files, you must extract the archive (restore the server) from an appropriate

©
5) If applications were manually activated and deployed or undeployed and deactivated

on the server to be replaced, activate and deploy or undeploy and deactivate them
again on the replacement server. This is done through the CLI. The following is an
example of activating and deploying BroadworksDms.
XSP_CLI/Maintenance/ManagedObjects> activate application BroadworksDms
21.0_1.551 /dms
XSP_CLI/Maintenance/ManagedObjects> deploy application /dms
12.2.3.5 Profile Server Replacement

1) Stop the BroadWorks application, replication, and the Configuration Agent on the
replacement server if it is running.
bwadmin@host$ repctl stop
bwadmin@host$ configctl stop
2) Re-mesh the SSH keys. On all peer servers, as root, run the following.
bwadmin@host$ rm -Rf /export/home/bwadmin/.ssh
As bwadmin, from either server, run the following.

bwadmin@host$ config-ssh -createKeys <other_peer_server>
3) Configure the replication peers on the new replacement server.
If the peer server is still in service, as bwadmin, run the peerctl ls command on
the existing peer server host to get a list of peers. Following is an example of a
bwadmin@host$ peerctl ls
HOSTNAME/ADDRESS
For example, if the server to be replaced has the host name ps1, the existing peer list
on the existing server should have one entry present set to “ps1”. However, that entry
may have a different host name defined, in which case you have to set it correctly.
Following is an example of the replacement server peerctl ls output.
-----------------------------------------------------------------------

©
backup/backup primary,unlocked
Given the previous examples, the following commands would be run on the
bwadmin@host$ peerctl add ps2
any directory on the replacement server. As the root user, run the following
commands.
the replacement server. Since the backup .tar file stores the relative paths of all
located when the backup was made, then perform the restore from the system root
from which you ran the tar xvf command.
NOTE: HTTPS certificates and keys are not backed up by the system backup. Information
stored in /var/broadworks/ssl/certs /var/broadworks/ssl/private should be backed up and stored
remotely whenever they are changed by the provider. The HTTPS certificate and keys must be
manually restored on the new server.
5) Start the Configuration Agent by running the following command.

7) Import the file repository files from the existing peer server, since they are not included
in the system backup. As the bwadmin user, run the following command on the
replacement server.
bwadmin@host$ rsync -vazurSHl <peerServer>:<file repository root>/. <file
repository root>/.
8) If applications were manually activated and deployed or undeployed and deactivated

on the server to be replaced, activate and deploy them or undeploy and deactivate
them again on the replacement server. This is done through the CLI. The following is
an example of activating and deploying DeviceManagementFiles.
PS_CLI/Maintenance/ManagedObjects> activate application
DeviceManagementFiles 21.0_1.551 /DeviceManagement
PS_CLI/Maintenance/ManagedObjects> deploy application /DeviceManagement
9) As the bwadmin user, run the following commands.

bwadmin@host$ repctl start

©
12.2.3.6 Database Server Replacement
For information on the Database Server replacement procedure, see the BroadWorks
Database Configuration Guide.
12.2.3.7 Element Management System Replacement
NOTE: The replacement server must be installed as a stand-alone server before proceeding
with the following steps.
1) Stop BroadWorks on the replacement server.

2) Re-mesh the SSH keys. On all peer servers, as root, run the following.
bwadmin@host$ rm -Rf /export/home/bwadmin/.ssh

bwadmin@host$ config-ssh -createKeys <bwadmin@other_peer_server>
4) Configure the replication peers on the new replacement server.
If the peer server is still in service, as bwadmin, run the peerctl ls command on
the existing peer server host to get a list of peers. Following is an example of a
PEER Role Status
========================================
ems01 PRIMARY NOTRUNNING
ems02 SECONDARY ACTIVE
For example, if the server to be replaced was the primary server, you must configure
the replacement server as primary with the same peer name. To configure the
redundancy, run the following command on the replacement server as bwadmin.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/setup-redundancy
If the new replacement server is the primary server, you need to reset the database
replication configuration on both servers by running the following command as
bwadmin. If the replacement server is the secondary server, the database replication
information is already reset.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/reset-redundancy.pl
5) Stop the Configuration Agent on the replacement server.


©
any directory on the replacement server. As the root user, run the following
commands.
This restores the last backup of the production server's configuration information to
the replacement server. Since the backup .tar file stores the relative paths of all
from which you ran the tar xvf command.
8) Import the database on the new replacement server. As bwadmin, run the following
command.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/importdb.pl <other server
hostname>
9) If the server to be replaced has LDAP enabled and was the primary server, restore
the LDAP database on the replacement server. The LDAP database was backed up
with the last BroadWorks system backup and it is located under
/var/broadworks/backup. As root, run the following command.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/bwLdapRestore.pl
<BackupFile>
12) Start the replication on both servers and start BroadWorks on the replacement server.
bwadmin@host$ repctl start
12.2.3.8 Execution Server Replacement

server.
commands.

©


©
13 Optional Remember Password Cookie for Administrators
System administrators can determine if the “Remember Password” function, in which a

cookie is set on the user’s PC, is available. The system provider can remove the
Remember Password check box from the login web page and refuse any existing user ID
Password cookies.
13.1 Modify Web.xml File

The optional cookie parameter is modified manually through the CLI. Since this parameter
is not replicated, the change needs to be propagated manually to all Application Servers
and Network Servers in a cluster. The setting is preserved during an upgrade.
13.1.1 Disable Cookie Support

1) Log in to the BroadWorks server using a telnet console.
2) Enter the CLI (bwcli) and set userCookieSupportEnabled to “false”.
NS_CLI/Applications/NSPortal/GeneralSettings> get
sslMode = on
loginAuthorizationLevel = all
userCookieSupportEnabled = true
NS_CLI/Applications/NSPortal/GeneralSettings> set
userCookieSupportEnabled false
*** Warning: The WebContainer application needs to be restarted for the
changes
to take effect ***
3) Restart the WebContainer application (restartbw WebContainer).
13.1.2 Enable Cookie Support

1) Log in to the BroadWorks server using a telnet console.
2) Enter the CLI (bwcli) and set userCookieSupportEnabled to “true”.
NS_CLI/Applications/NSPortal/GeneralSettings> get
sslMode = on
loginAuthorizationLevel = all
userCookieSupportEnabled = false
NS_CLI/Applications/NSPortal/GeneralSettings> set
userCookieSupportEnabled true
*** Warning: The WebContainer application needs to be restarted for the
changes
to take effect ***
3) Restart the WebContainer application (restartbw WebContainer).

©
14 Install BroadWorks License Files
The BroadWorks licensing files are copied to the proper BroadWorks directories. Note
that your BroadWorks license includes software from other companies. For details,
consult your license agreement from BroadSoft.
The software CD is ejected and the user is prompted to enter the license CD into the CD
drive. If the BroadWorks licensing files are not located on a CD, then the directory path
containing the license files can be entered instead. Note that the root password is
required for the license file installation.
Follow these steps:
1) su to root and launch the install-license.pl script.
2) # cd /cdrom/cdrom0/<BW release>/install.
3) # ./install-license-pl <BW release>. The following response appears
(sample provided).
-----------------------------------------------------
Install license files to NS_Rel_14.0_1.538
Please insert your license CD into the CDRom drive

Wait about 20 seconds and press Enter
If your license files are not located on a CD, enter the

directory path where you put them
Hit 'q' if you do not have a license CD or wish to skip this step,
Installing BroadWorks license files to /usr/local/broadworks/

NS_Rel_14.0_1.538/conf from /cdrom/cdrom0/
Done
The license installer script automatically installs the BroadWorks license files license.sig
and license.txt in /usr/local/broadworks/<BW version>/conf.
However, in the event that the license installer script would either not be called or would
not be successful, the following steps should be executed as root.
4) Log in as (or su to) root.
5) Copy the BroadWorks license files (license.txt and license.sig) from the license-
backup directory to the configuration directory.
# cd /export/home/bwadmin/license-backup
# cp license.* /usr/local/broadworks/<BW release>/conf
6) After copying the license files in the conf directory, you need to activate it. For
Network Servers and Application Servers, log in to the CLI and execute the set
command under /System/Licensing.
AS_CLI/System/Licensing> set
For all other BroadWorks servers, restart the BroadWorks applications.

bwadmin@host$ restartbw

©
15 Uninstall Obsolete BroadWorks Versions
As BroadWorks servers are upgraded, old BroadWorks versions are not deleted but they
remain on the server. BroadSoft recommends that after an upgrade, the current active
version and the last version should be kept on the server (at a minimum). Older
BroadWorks versions can be uninstalled using the following procedure.
To uninstall an obsolete server version (for example, NS_Rel_19.0_1.131 on the Network
Server):
1) Log in (or su -) to root.
2) Change the directory (cd) to
/var/broadworks/<current_release>/uninstall and run the uninstall-
bwserver.pl script.
3) Type # /uninstall-bwserver.pl -r. The “-r” option directs the script to
remove the corresponding /var/broadworks release directory. The following response
#./uninstall-bwserver.pl -r
Select which release to remove (0 to stop the process)
0 - none
1 - NS_Rel_11.0_1.131
2 - NS_Rel_12.0_1.411
3 - NS_Rel_13.0_1.362
4 - NS_Rel_14.0_1.606
Enter selection [0] ?1
You are about to remove broadworks release NS_Rel_11.0_1.131
Do you want to continue? (y/n) [n]?y
Un-Installing BW Server, version NS_Rel_11.0_1.131...

Removing /var/broadworks/NS_Rel_11.0_1.131

©
16 Uninstall Obsolete Third-Party Components
As BroadWorks servers are upgraded, third-party components are also upgraded. It is not
typically required to uninstall these old components. However, it might be interesting to
get rid of old software binaries as they become obsolete when free disk space becomes
an issue.
To uninstall an obsolete third-party component, the specific./uninstall-
<component>.pl script must be used.
1) Log in (or su -) to root.
2) Change the directory (cd) to /var/broadworks/<current_release>/uninstall and run the
./uninstall-<component>.pl script.
3) Select the version of the component to remove and confirm operation.
[root@as1.example.org /] cd
/var/broadworks/AS_Rel_22.0_1.1123/uninstall
[root@as1.example.org uninstall]# ./uninstall-java.pl

Select which java version to remove (0 to cancel)
0 - none
1 - jdk1.7.0_99
2 - jdk1.8.0_102b
Enter selection [0] ?1
You are about to remove java version jdk.1.7.0_99
Un-Installing Java, version jdk.1.7.0_99...

[root@as1.example.org uninstall]#
NOTE: These scripts run some basic verifications to ensure that the selected third-party
component version is not currently used and can be safely deleted. However, removing the
previous version of a component might prevent future attempts to rollback BroadWorks from
succeeding if the target release still requires it.
Alternatively, these scripts can be invoked with an argument specifying the version to
remove.
1) Invoke the script with a version argument.
[root@as1.example.org /] cd /var/broadworks/AS_Rel_22.0_1.1123/uninstall
[root@as1.example.org uninstall]# ./uninstall-java.pl jdk.1.7.0_99

You are about to remove java version jdk.1.7.0_99
Un-Installing Java, version jdk.1.7.0_99...
[root@as1.example.org uninstall]#

©
17 Change IP Address, Host Name, FQDN Name, Gateway, or DNS Server
This section describes the maintenance steps to be performed on a BroadWorks server

when changing an IP address, host name, FQDN name, gateway, or DNS server.
Following are the steps, on a per-server basis, that must be executed when changing a
BroadWorks server IP address, host name, or FQDN name, gateway, or DNS server:
Application Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Restart BroadWorks Software Manager.
4) Update the BroadWorks network configuration.
5) Re-configure redundancy routing, if required.
6) Update the Application Server call processing routing information.
7) Update the Application Server address information on the Network Server, if required.
8) Update the Application Server address information on the Open Client Server (OCS)
application, and Xtended Services Platform, if required.
9) Re-run the EMS auto-discover process, if applicable.
Element Management System:

1) Stop BroadWorks.
1) Stop replication.
5) Update EMS redundancy configuration, if applicable.
6) Start replication.
7) Import the EMS database, if applicable.
Media Server:
1) Stop BroadWorks.
5) If deployed with the Network Server Media Server Selection policy:
6) Update the Media Server address on the Network Server, if required.
7) If NOT deployed with the Network Server Media Server Selection policy:
Update the Media Server address on the Application Server, if required,

©
Network Server:
1) Stop BroadWorks.
5) Re-configure redundancy routing, if required.
6) Update the Network Server routing alias list, if required.
8) Update the Network Server address information on the Application Server, Open
Client Server application, and Xtended Services Platform, if required.
Xtended Services Platform:

1) Stop BroadWorks.
5) Update the Application Server or Profile Server access lists (OCI and ExtAuth) with
the Xsp address, if required.
6) Rerun the EMS auto-discover process, if applicable.
Profile Server:
1) Stop BroadWorks.
5) Update the Profile Server HTTP Server configuration, creating new HTTP servers for
the new IP and deleting old ones. This is done from the
PS_CLI/Interface/Http/HttpServer CLI level. For more information, see the
BroadWorks Profile Server Configuration Guide.
6) Update the Profile Server HTTP Alias with the new host name as required. This is
done from the PS_CLI/Interface/Http/HttpAlias CLI level. For more information, see
the BroadWorks Profile Server Configuration Guide.
7) Update the Profile Server address information on the Xtended Services Platform, if
required.
8) Update the notification subsystem configuration, if required.
9) Update the Profile Server address information on the Network Server, if required.

©
Execution Server:
1) Stop BroadWorks.
5) Update the notification subsystem configuration.
6) Update the OCI provisioning network access list on the Profile Server.
7) Update the Execution Server address information on the Network Server, if required.
Messaging Server:
1) Stop BroadWorks.
Sharing Server:
1) Stop BroadWorks.
WebRTC Server:
1) Stop BroadWorks.
Database Server:
NOTE 1: IP address/host name configuration on the Database Server depends on the

deployment model and as a result, requires extra care. Changing the Database Server host

©
name is only possible in single instance deployments; changing the host names of a locally
redundant cluster configuration is not supported.
NOTE 2: In a geographically redundant configuration, the host name change must be done
while the site status is “standby” and while the DBSObserver is shut. Host name modification of
the primary site is not supported.
To change the Database Server host name, the following sequence must be executed.
1) On the Profile Server, stop the DBSObserver (if configured).
bwadmin@ps$ stopbw DBSObserver
2) On the Database Server, create a hostname alias using the new host name. This can
be done by adding a CNAME record to your DNS or by adding the new host name to
the host name line in /etc/hosts, as shown in the following example.
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.1.101 newname.example.com oldname.example.com
NOTE: This alias must be available to all sites in the redundant configuration and must be
defined before the old name; otherwise, during reconfiguration the old name is still used.
3) While keeping BroadWorks running, run the following command and follow the
instructions it provides.
bwadmin@oldname$ sitectl config hostname=newname.example.com
Checking parameters... [DONE]
WARNING: Reconfiguring the hostname on the Database Server requires some

preparation steps.
Refer to the BroadWorks Maintenance Guide for details.
Do you still want to proceed? (y/n) [y]?y

Checking hostname... [DONE]
Checking host reachability... [DONE]
Checking host alias... [DONE]
Checking ssh meshing... [DONE]
Checking oracle env... [DONE]
Checking deployment type... [SI]
Checking redundancy... [DONE]
Checking database readiness ... [DONE]
Getting ASM disk groups... [DONE]
Getting ASM spfile... [DONE]
Getting primary db name... [DONE]
Getting current listener port number... [DONE]
Updating database listener... [DONE]
Updating listener host config... [DONE]
Updating database TNS ... [DONE]
Cleaning up database TNS on oldname.example.com... [DONE]
Deconfiguring Oracle Grid... [DONE]
The new host name must now be activated.
Please login as 'root' and set the new system hostname using 'hostname

©
newname.example.com'.
During this time, the host must be accessible using both
oldname.example.com and newname.example.com.
Once done, press <enter> to continue.
Configuring Oracle Grid for new hostname... [DONE]

Adding basic Oracle Grid services... [DONE]
Registering database 'bwCentralizedDb1' with Oracle Grid... [DONE]
Adding service 'bwCentralizedDb' for database... [DONE]
Disabling fast_start failover... [DONE]
Changing protection mode... [DONE]
Disabling data guard db... [DONE]
Update data guard db host... [DONE]
Re-enabling data guard db... [DONE]
Restoring protection mode ... [DONE]
Re-enabling fast_start failover... [DONE]
The database is now configured to use the new host name
'newname.example.com .
You must now restart BroadWorks, refer to the BroadWorks Maintenance
Guide for details.
4) Stop BroadWorks.
bwadmin@newname$ stopbw
5) Drop the old host name from the name service configuration, keeping only the new
name.
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.1.101 newname.example.com oldname.example.com

bwadmin@newname$ startbw
8) Update the Database Server address information on the Profile Server and Execution
Server, if required.
9) Start the DBSObserver, if applicable.
10) Manually add the Database Server to the EMS configuration.
17.1 Configure IP Networking Information at UNIX Level
NOTE: Read the introduction section before executing the steps described in this section, as
other actions may be required.
The following steps are required:

1) Log in as or su to root.
2) Change the following files: /etc/hosts (IP address on line that includes loghost).
If you are modifying IP addresses for dual homing, IP multipathing, or redundant
networking, those IP addresses must also be changed here.

©
If changing the host name, these files must also be modified.
/etc/nodename
/etc/hostname.hme#, where hme# is the network interface device. This may be

dmfe# or qfe# on some systems, depending on the Ethernet hardware available.
There are multiple files for the system configured to use multiple interfaces (for
example, hostname, hme0, hostname.hme1, and so on).
/etc/net/ticlts/hosts
/etc/net/ticots/hosts
/etc/net/ticotsord/hosts
17.2 Restart BroadWorks Software Manager

You must restart the BroadWorks Software Manager.
MTLNS04$ stopswman.pl
Stopping SW Manager version: 635548
MTLNS04$ startswman.pl
Starting SW Manager version: 635548

©
17.3 Update BroadWorks Network Configuration
17.3.1 Restart the configuration agent

Prior to update the HTTP server network configuration, you must restart the configuration
agent.
MTLNS04$ configdctl restart
Restarting configd...Setting system info values
Patching configuration...
[ok]
[ok]
17.3.2 HTTP Server Network Configuration

The HTTP network configuration on a BroadWorks server is all done through the CLI. The
levels under CLI/Interface/Http are used for such configuration. The httpServer level
allows the addition of interfaces and ports on which the HTTP server is listening. The
following is an example.
NS_CLI/Interface/Http/HttpServer> get
Interface Port Name Secure Cluster
FQDN
=========================================================================
================
192.168.12.66 443 mtl64lin12.mtl.broadsoft.com true
mtl64lin12.mtl.broadsoft.com
192.168.12.66 80 mtl64lin12.mtl.broadsoft.com false
mtl64lin12.mtl.broadsoft.com
2 entries found.
The httpAlias level allows the definition of host alias for a given interface.
NS_CLI/Interface/Http/HttpAlias> get
Interface Alias Cluster FQDN
========================================================
192.168.12.66 myAlias.mtl.broadsoft.com
1 entry found.
Finally, the HttpBinding level allows the specification as to which web application is bind on
which interface. This functionality can be used to restrict the availability of a web
application on a specific interface. By default, httpBinding is empty, allowing any web
application to run on any interface/port combinations.
NS_CLI/Interface/Http/HttpBinding> get
Interface Port Application Name
=======================================
192.168.12.66 80 NSPortal
1 entry found.
17.3.3 SNMP Trap Source Address Configuration

The trap source address configuration on a BroadWorks server is done through the CLI.
The level CLI/Interface/SNMP/Agent is used for such configuration.

©
AS_CLI/Interface/SNMP/Agent> get
port = 8001
encoding = ISO-8859-1
readCommunity = public
writeCommunity = public
trapCommunity = public
trapSourceAddress = 10.9.0.162
disableV2 = false
17.4 Configure Redundancy Routing
NOTE: Read the introduction section prior to executing the steps described in this section, as
The Application Server, Network Server, and Messaging Server redundancy routing
configuration have to be modified when changing a server host name or IP address. This
is done from the UNIX prompt, by executing the peerctl command.
It is important to execute the commands in the following order:
1) Stop replication on all servers of the cluster using Repctl stop.
2) Execute the peerctl command on all servers of the cluster.
The following is an example of the peerctl command.
MTLNS04$ peerctl
Usage:
(When the hostname is optional, omitting it means the current peer)
[-dsn <dsn>] ls
[-dsn <dsn>] add <hostname> <address> <repType(0=master;1=slave)>
[-dsn <dsn>] set <originalhostname> <hostname> <address>
<repType(0=master;1=slave)>
[-dsn <dsn>] del <hostname>
[-dsn <dsn>] lock [<hostname>|all]
[-dsn <dsn>] unlock [<hostname>|all]
[-dsn <dsn>] setPrimary [<hostname>]
[-dsn <dsn>] waitAllLocked <timeout in minutes (typically 12)>
NOTE: Locking a server can be almost immediate or may take up to 10+
minutes
depending on server activity.
MTLNS04$ peerctl set mtlnsold mtlnsnew 192.168.12.70 repType 0
3) Restart replication on all servers of the cluster, using Repctl start.

4) Note that, for an Application Server, it may be required to re-import that database on
the secondary server, in cases such as when the replication is stopped for an
extended period.
17.5 Update Application Server Call Processing Routing Information

Some start-up parameters may have to be modified for the SIP and MGCP signaling to
function properly after changing the address of an Application Server. These parameters
can be modified (if present) under the System/StarupParm level.
Parameter Description

©
Parameter Description
publicIPAddress This is the default IP address used by the Application Server for the
SIP stack and other applications. On a server with a single NIC, this is
the only parameter configured.
privateIPAddress On multi-homed server, this is used to specify the IP address to use on

the access side.
bw.sip. On multi-homed server, this is used to specify the IP (interface) to use

accessinterfaceviahost on the access side. This parameter takes precedence over the
privateIPAddress parameter for SIP signaling.
bw.sip. On multi-homed server, used to specify the IP (interface) to use on the

networkinterfaceviahost network side. This parameter takes precedence over the
publicIPAddress parameter for SIP signaling.
NOTE: It is important to validate and change (if required) the parameters listed above whenever
an IP address or host name is changed for an Application Server.
17.6 Update Application Server IP Address on Network Server
NOTE: Read the introduction section before executing the steps described in this section, as
The Application Server IP address could be configured on the Network Server. After
making changes to the IP address on the Application Server, the Network Server may also
need to be updated. If the Application Server address on the Network Server is an FQDN
or an alias, this step can be skipped.
1) Log in to the Network Server command line interface.
2) Go to the NS_CLI/System/Device/HostingNE/Address> level.
3) Add the new IP address of the Application Server.
4) Delete the old IP address of the Application Server.
The following is an example of updating the Application Server IP address on the
Network Server.
NS_CLI/System/Device/HostingNE/Address> g
Retrieving data… Please wait…
HostingNe NodeID Address type cost weight port
transport
=======================================================================
==
AS1 0 192.168.12.6 DualRouting 1 99 5060
tcp
1 entry found.
NS_CLI/System/Device/ResourceNE/Address> add AS1 0 192.168.12.7
DualRouting 1 99 5060 tcp
…Done
NS_CLI/System/Device/ResourceNE/Address> delete AS1 0 192.168.12.6

…Done
NS_CLI/System/Device/HostingNE/Address> g

©
HostingNe NodeID Address type cost weight port
transport
=======================================================================
==
AS1 0 192.168.12.7 DualRouting 1 99 5060
tcp
1 entry found.
17.7 Update Network Server IP Address on Application Server
The Network Server IP address can be configured on the Application Server. After
making changes to the IP address on the Network Server, the Application Server may also
need to be updated. If the Network Server address on the Network Server is an FQDN or
an alias, this step can be skipped.
1) Log in to the Application Server command line interface.
2) Go to the AS_CLI/System/Device/NetworkServers/Routing level.
3) Add the new IP address of the Network Server.
4) Delete the old IP address of the Network Server.
The following is an example of updating the Application Server IP address on the
Network Server.
AS_CLI/System/Device/NetworkServers/Routing> get
Net Address Port Transport Poll OpState Description
=========================================================================
192.168.12.80 5060 UDP false enabled MTLNS01-DNSlong
1 entry found.
AS_CLI/System/Device/NetworkServers/Routing> add 192.168.12.70 tcp
description “MTLNS04-DNSlong” port 5060
…Done
AS_CLI/System/Device/NetworkServers/Routing> delete 192.168.12.80

…Done
Net Address Port Transport Poll OpState Description

=========================================================================
192.168.12.70 5060 UDP false enabled MTLNS01-DNSlong
1 entry found.

©
17.8 Update Network Server Routing Alias List
The Network Server aliases may need to be updated when modifying a Network Server IP
address.
2) Go to the NS_CLI/System/Alias level.
3) Add the new address or domain to the Network Server alias list.
4) Delete the old IP address or domain from the Network Server alias list.

©
The following is an example of updating the Network Server alias IP address on the
Network Server.
NS_CLI/System/Alias> get
127.0.0.1
192.168.13.80
NS_CLI/System/Alias> add 192.168.12.70
…Done
NS_CLI/System/Alias> delete 192.168.12.80

…Done
NS_CLI/System/Alias> get
127.0.0.1
192.168.13.80
17.9 Update Media Server IP Address on Network Server
A Network Server for Media Server Selection can also use the Media Server IP address.
After making changes to the IP address on the Media Server, the Network Server needs
to be updated as well.
2) Go to NS_CLI/System/Device/ResourceNE/Address level.
3) Add the new IP address of the Media Server.
4) Delete the old IP address of the Media Server.
The following is an example of updating the Media Server IP address on the Network
Server.
NS_CLI/System/Device/ResourceNE/Address> g
Resource NE Address Cost Weight Port Transport
MS1 192.168.12.6 1 10 5679 tcp
NS_CLI/System/Device/ResourceNE/Address> add MS1 192.168.12.7 1 10 5679

tcp
…Done
NS_CLI/System/Device/ResourceNE/Address> delete MS1 192.168.12.6

…Done
NS_CLI/System/Device/ResourceNE/Address> g
Resource NE Address Cost Weight Port Transport
MS1 192.168.12.7 1 10 5679 tcp
1 entry found.

©
17.10 Update Media Server IP Address on Application Server
The Media Server IP address is most likely being used by an Application Server to stream
media. After changing the IP address on the Media Server, the Application Server needs
to be updated as well.
1) Log in to the Application Server command line interface.
2) Go to the AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice
level.
3) Add the new IP address of the Media Server.
4) Delete the old IP address of the Media Server.
The following is an example of updating the Media Server IP address on the
Application Server.
AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice> get
Net Address Port Transport Description
===========================================
192.168.1.1 5060 TCP
1 entry found.
AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice>add
192.168.1.2 tcp port 5060
…Done
AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice>delet
e 192.168.1.1
…Done
AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice> get
Net Address Port Transport Description
===========================================
192.168.1.2 5060 TCP
1 entry found.
17.11 Update Application Server and Network Server Address on OCS
When updating the Application Server and/or Network Server IP address, the
Communication Utility IP address on the Application Server, Xtended Services Platform,
and/or EMS (if applicable) must be updated.
1) Log in to the BroadWorks command line interface.
2) Go to the System/CommunicationUtility/DefaultSettings level.
3) Depending on the mode, change the IP address of either the primary or secondary
Application Server or the Network Server.

©
The following is an example of an update of an Application Server IP address for the
Communication Utility.
AS_CLI/System/CommunicationUtility/DefaultSettings> get
AS:
asPrimaryAddress=localhost
asSecondaryAddress=
asOCIPort=2220
asOCISecurePort=2320
asOCICPort=2221
asOCICSecurePort=2321
provisionOnSecondary=false
reconnectionTimerSecs=30
responseTimeoutSecs=10
globalTransactionLimit=100
userTransactionLimit=5
transactionLimitPeriodSecs=1
useSecureBCCT=false
allowExtAuthAgent=false
AS_CLI/System/CommunicationUtility/DefaultSettings> set mode AS

192.168.12.5 asSecondaryAddress 192.168.12.6
*** Warning: BroadWorks needs to be restarted for the changes to take

effect ***
17.12 Update Xtended Services Platform IP Address on Network Server
NOTE: Read the introduction section prior to executing the steps described in this section, since
The Xtended Services Platform address must be properly configured for the Location
Server function on the Network Server to work. After making changes to the IP address
on the Xtended Services Platform, the Network Server must be updated as well.
2) Go to the NS_CLI/System/Device/XSPServerFarm/Address level.
3) Add the new IP address of the Xtended Services Platform.
4) Delete the old IP address of the Xtended Services Platform.
The following is an example of an update of the Xtended Services Platform IP
address on the Network Server.
NS_CLI/System/Device/XSPServerFarm/Address> get
XSPServerFarm NodeID Address type cost weight
============================================================
WSF 0 192.168.12.16 Access 1 50
WSF 0 mtlws01 Alias - -
WSF 1 192.168.12.18 Access 1 50
4 entries found.
NS_CLI/System/Device/XSPServerFarm/Address> add WSF 1 192.168.23.12

Access 1 50
...Done

©
NS_CLI/System/Device/XSPServerFarm/Address> del WSF 1 192.168.12.18
...Done
NS_CLI/System/Device/XSPServerFarm/Address> get
XSPServerFarm NodeID Address type cost weight
============================================================
WSF 0 192.168.12.16 Access 1 50
WSF 1 192.168.23.12 Access 1 50
4 entries found.
17.13 Update Network Server or Application Server Address on Xtended Services

Platform
The BwCommunicationUtility component on the Xtended Services Platform requires

knowledge of the Network Server or Application Server depending on the system setup.
For more information, see the BroadWorks Xtended Services Platform Configuration
Guide.
17.14 Update EMS Redundancy Configuration
NOTE: Read the introduction section prior to executing the steps described in this section as
On a redundant EMS system, redundancy information such as the peer host name and
address need to be modified to represent the new environment. Run the update-
redundancy.pl script from the UNIX prompt on all peers.
./bwUpdatePeerInformation.pl
Usage: bwUpdatePeerInformation.pl oldValue newValue
This script is used when the hostname or the IP address has changed for
one peer.
Parameters
-------
oldValue - The previous value.
newValue - The new value.
Example:
./ bwUpdatePeerInformation.pl ems01 ems02
Updating local peer address.........[Ok]
Updating peer hostname..............[Ok]
Updating peer address...............[Ok]
Updating db replication info........[Already up-to-date]
Updating privileges.................[Ok]

©
17.15 Import EMS Database
On the redundant EMS system, the database must be synchronized on all peers to make
sure that the replication is not lagging. If an IP address has changed, the database may
no longer be synchronized.
Import the database on the peer that has changed their IP or host name (using the
importdb.pl command from the UNIX prompt).
17.16 Update Notification Subsystem Configuration

For the Execution Server as well as the Provisionning application on the Profile Server the
notification subsystem configuration need to be updated after the IP address has been
changed. The address can be changed from the CLI under the following contexts:
PS_CLI/Applications/Provisioning/Database/NDS/Notifications
PS_CLI/Applications/Provisioning/SubscriberAgent/Database/NDS/Notifications
XS_CLI/Applications/ExecutionDataless/Database/NDS/Notifications
PS_CLI/Applications/Provisioning/Database/NDS/Notifications> get
address = 10.9.0.63
port = 2450
secure = false
notificationDuration = 5
subscriptionDuration = 1440
17.17 Update OCI Provisioning Network Access List on Profile Server

When changing the IP address of the Execution Server, the OCI provisioning network
access list on the Profile Server for the provisioning application needs to be changed. See
the BroadWorks XS Mode Configuration Guide for more details.
17.18 Update Execution Server IP Address on Network Server

When changing the IP address of the Execution Server, the Network Server configuration
needs to be updated. Refer to the BroadWorks XS Mode Configuration Guide for more
details.
17.19 Update Profile Server IP Address on Network Server

When changing the IP address of the Profile Server where the Provisioning application is
deployed, the Network Server configuration needs to be updated. See the BroadWorks
XS Mode Configuration Guide for more details.

©
18 Change BCCT Listening Ports
This section describes the maintenance steps to be performed on a BroadWorks server

when the BroadWorks Common Communication Transport (BCCT) listening ports
XSListeningPort, PSListeningPort, XSSecureListeningPort, and PSSecureListeningPort
are changed on the Application Server or on the Network Server.
The BCCT listening ports are configurable from the AS_CLI/
/Interface/CommonCommunicationTransport level or from the NS_CLI/
/Interface/CommonCommunicationTransport level.
It is strongly recommended to change the BCCT listening ports of all Application Servers
and Network Servers to use the same values for the BCCT listening ports.
Following are the steps that must be executed after the BCCT listening ports are changed,
on a per-server basis:
Element Management System:

1) Update the BCCT port number used by the cut-through session functionality to the
new value of “PSListeningPort” or “PSSecureListeningPort” for each server being
modified. This is done from the EMS Web Client. (For more information, see the
BroadWorks Element Management System User Guide Part 2.)
2) Update the BCCT port number used by the Auto-Discovery functionality to the new
value of “PSListeningPort” or “PSSecureListeningPort”. This is done from the EMS
Web Client. (For more information, see the BroadWorks Element Management
System Administration Guide, Part 1.)
Network Server:
1) Exit all CLIs.
Xtended Services Platform:

1) Update the Xsp communication utility default settings and change the value of the
asOCIPort and asOCISecurePort corresponding to the new value of
“PSListeningPort” and “PSSecureListeningPort” respectively. This is done from the
XSP_CLI/ /System/CommunicationUtility/DefaultSettings CLI level. For more
information, see the BroadWorks Xtended Services Platform Configuration Guide.

©
19 Solaris DiskSuite Maintenance
19.1 Background
During installation and configuration of disk mirroring using Solstice DiskSuite, the Mdlogd
daemon has been set up to report problems via SNMP traps, in addition to logging them to
/var/adm/messages. These traps, in conjunction with use of DiskSuite status commands,
provide background information for maintenance actions to be taken. This section
describes the more common maintenance and administration tasks related to the mirrored
configuration used by BroadWorks. For more information, see the Solstice DiskSuite
Users Guide and the Solstice DiskSuite Reference Guide.
Solstice DiskSuite provides commands to track maintenance states and to administer
mirrors and submirrors. It also provides commands to track states and to administer
metadatabase replicas.
19.1.1 Submirror Status

The metastat command is used to determine the state of mirrors and submirrors. There
are three basic states for submirrors: okay, maintenance, and last erred.
When there is a problem with a submirror, it is placed in the maintenance state, for
example, for a media error, an I/O error, or a hardware problem. However, if this happens
then there is another submirror with data integrity. For example, when one submirror is
corrupted, it is placed in the maintenance state and it is therefore out of synchronization.
However, another submirror in the mirror is functioning normally and it would be in the
okay state.
A submirror is placed in the last erred state when there is a problem with the submirror
while the other submirror is already in the maintenance state. Therefore, if a submirror
enters the last erred state, some degree of data integrity has been compromised. When a
problem arises with a mirror, the submirror in maintenance state must be addressed
before the submirror in last erred state can be addressed.
Following are a few examples using the d14 (/bw) mirror with submirrors d24 and d34:
 Submirror d34 experiences write errors and goes into the maintenance state.
Submirror d24 does not experience any errors and remains in the okay state. In this
case, submirror d34 can be “fixed”, after which mirror d14 is in the okay state.
 Submirror d34 experiences write errors and goes into the maintenance state. Before
corrective action is taken, submirror d24 experiences write errors. Since d34 was
already in the maintenance state, d24 goes into the last erred state. In this case,
submirror 34 must be “fixed” before submirror 24 can be “fixed”.
There are various reasons for a submirror to transition from an okay state. Possible
causes include, but are not necessarily limited to, transitory problems, for example, power
loss, disk I/O errors, bad sectors, physical removal, and re-insertion of the same disk, as
well as a faulty disk. Taking corrective action to get the state of the mirrors/submirrors
back to the okay state is part of regular maintenance.
19.1.2 Replacement Disk Requirements

In the event a disk in a mirrored pair must be replaced, it must be replaced with a disk that
has the same geometry as its peer submirror. That is, the replacement disk must be
partitioned the same way as its peer submirror.

©
19.1.3 Metadatabase Replica Status
The metdb -i command is used to determine the state of metadatabase replicas. The
system maintains multiple copies of the replicas (the BroadWorks Software Management
Guide recommends three copies on each disk). The system uses a majority consensus
algorithm to determine whether there is overall integrity in the metadatabases. Upon
booting, the system checks whether the number of good copies of the metadatabase
comprises 51% or more of the number of metadatabase copies. If so, booting is allowed
to proceed. Otherwise, booting is stalled. There are various reasons for faulty
metadatabase replicas or failure to have a majority consensus on replicas. Possible
reasons include, but are not necessarily limited to, transitory problems (for example, power
loss, disk I/O glitches, or bad sectors), faulty disk, and accidental deletion of replicas.
Repairing bad copies of the metadatabase is part of regular maintenance.
19.2 Maintenance and Recovery Procedures
19.2.1 System Running
19.2.1.1 Transitory Submirror Problems

A submirror can transition out of the okay state for various reasons (for example, power
loss, disk I/O glitches, bad sectors, physical removal, and re-insertion of the same disk).
To correct the situation, repair the faulty sectors, if necessary (depending on the extent of
the problem, you may have to perform disk repair procedures as documented in the
Solaris 9 Administration Guides) and re-enable the slice. It causes a re-sync of the
submirror. The -e option is for enabling.
If necessary, perform disk repair procedures as documented in the Solaris 8
Administration Guides. However, note that this may incur downtime and it should be
performed during a maintenance window.
<# metareplace -e <mirror metadevice name> <faulty slice name>
The following example shows re-enabling of the /bw slice on the secondary disk.
#metareplace -e d14 c0t1d0s4
19.2.1.2 Transitory Database Replica Problems

Faulty database replicas (as reported when invoking metadb -i) can be repaired by
repairing faulty sectors (depending on the extent of the problem, you may have to perform
disk repair procedures, as documented in the Solaris 9 Administration Guides), removing
the replica(s), and re-adding them. However, never remove all replicas from a system.
Always make sure that there is 51% or more replicas in good condition, so that the system
is bootable.
If bad replicas exist on only one disk, then:
1) If necessary, perform disk repair procedures as documented in the Solaris 8
Administration Guides. However, note that this may incur downtime and it should be
performed during a maintenance window.
2) Remove the faulty state databases from the partition holding them (metadb -d <slice
name containing bad databases>).
3) Re-add state databases to that partition (metadb -a -c 3 <same slice name as
above>).
4) Verify the re-added state databases are okay (metadb -i).

©
If bad replicas exist on the metadb partitions of both disks, then:
1) Follow the instructions above for one disk.
2) Repeat above procedures for the other disk.
The following is an example. It uses the same partitioning and mirroring scheme as
documented in the fresh installation section of the BroadWorks Software Management
Guide. In this example, there is a bad database replica on the secondary disk.
# metadb -d c0t1d0s3
# metadb -a -c 3 c0t1d0s3
# metadb –i
19.2.1.3 Faulty Secondary Disk

The following procedure is used if it has been determined that the secondary disk itself, is
faulty and cannot be repaired (for example, via the format tool or other procedures
documented in the Solaris 9 Administration Guide). Perhaps there are many recurring
problems, or a metastat and metadb -I indicate all submirrors and metadatabase replicas
on that disk are faulty, or it is obvious by other means that the disk simply does not
function, the disk must be replaced, and the mirror re-constructed. Once a replacement
disk is available, perform the following steps during a maintenance window:
1) Remove state databases from the secondary disk (metadb -d <slice name of metadb
partition on secondary disk>).
2) If after removing these state databases, you are left with less than 51% state
databases in good shape (as seen by the metadb -I command), add more to the
metadb partition on the good disk (metadb -a -c <number to add> <metadb slice
name on boot disk>). This makes sure you can successfully boot up the server.
3) Detach all submirrors on the problem disk by issuing metadetach commands
successively for root, swap, and /bw submirrors (for each submirror on secondary
disk, metadetach -f <mirror> <submirror>).
4) Boot to single user mode (stop BroadWorks, sync, halt, boot -s).
5) Physically replace the problem disk.
6) Repartition the new disk with the same slice information as the boot disk (prtvtoc <raw
device name of boot disk> pt, fmthard -s pt <raw device name of secondary disk>).
7) Re-add state databases to the new disk (metadb -a -c 3 <slice name of metadb
partition on secondary disk>).
8) Reboot (reboot).
9) Re-attach all submirrors on the replaced disk by issuing the metattach command
disk, metattach <mirror> <submirror>). This causes the submirrors on the secondary
disk to re-sync.
Guide. In this example, assume that two more metadatabase replicas must be added to
the boot disk to satisfy the 51% consensus algorithm. The example would have to be
adjusted if the partitioning and mirroring scheme is different. That is, when the
replacement disk is available or when the primary disk can be repaired, and when
maintenance window available).

©
# metadetach -f d10 d30
<stop broadworks>
# sync
# halt
ok boot -s
<physically insert new disk>
# prtvtoc /dev/rdsk/c0t0d0s2 > pt
# fmthard -s pt /dev/rdsk/c0t1d0s2
# reboot
# metattach d10 d30
# metattach d11 d31
# metattach d14 d34
19.2.1.4 Faulty Primary Disk

If it has been determined, that the primary disk itself is faulty and it cannot be repaired, for
example, via the format tool or other procedures documented in the Solaris 9
Administrative Guide. (Perhaps there many recurring transitory problems, or a metastat
and metadb -I indicate that all submirrors and metadatabase replicas on that disk are
faulty.) On the other hand, if it is obvious by other means that the disk simply does not
function, the disk must be replaced, and the mirror re-constructed.
Until a replacement disk is available, perform any necessary reboots off the secondary
disk (the system does not boot if the boot disk at the boot PROM level is set to the primary
disk).
Once a replacement disk is available, perform the following steps during a maintenance
window.
1) Delete the failed metadatabase replicas from the primary disk (metadb -d <primary
disk metadb slice name>).
2) Boot to single user mode on the secondary disk (stopbw, sync, halt, boot <diskn> -s).
3) Physically replace the problem disk with the replacement disk.
4) Repartition the new disk with the same slice information as the secondary disk
(prtvtoc <raw device name of secondary disk> pt, fmthard -s pt <raw device name of
boot disk>).
5) Re-add metadatabase replicas to the primary disk (metadb -a -c 3 <primary disk
metadb slice name>).
6) Reboot off the secondary disk (stop broadworks, sync, halt, boot <diskn>).
7) Re-enable all submirrors on the primary disk by issuing the metareplace command
successively for root, swap, and /bw submirrors (for each submirror on the primary
disk, metareplace -e <mirror metadevice name> <faulty slice name>). This
synchronizes the submirrors on the primary disk with the submirrors on the secondary
disk.
8) Once re-synchronized, you can stop BroadWorks and reboot from the original device
(reboot) if desired.

©
Guide. Although it may vary from platform to platform, the primary disk (boot disk) is
typically disk0, and the secondary disk is disk1. The example has to be adjusted if the
partitioning and mirroring scheme is different.
<until replacement disk is available, any and all reboots should be
performed off the secondary disk>
<when replacement disk is available or primary disk can be repaired, and
maintenance window available>
<stop broadworks>
# sync
# halt
ok boot disk1 -s
<physically replace primary disk with new disk>
<stop broadworks>
# sync
# halt
ok boot disk1
# metareplace -e d10 c0t0d0s0
<verify the sync operation has been completed>
<perform following steps if verification of ability to boot off primary
disk is desired>
<stop broadworks>
# reboot
19.2.2 System Fails to Boot Properly
19.2.2.1 Primary Disk is Faulty

In the event that the primary disk is faulty and the server is rebooted without having
repaired the primary disk, the boot sequence fails. Typically, on the console there is the
message, “cannot open boot device”. The recovery procedure is to attempt to boot off the
secondary drive temporarily, until a replacement disk is available, at which time steps can
be taken during a maintenance window to restore the mirror.
1) From the ok boot prompt, boot from the secondary disk (boot <diskn>).
2) Depending on how many metadatabase replicas are faulty, the boot of the secondary
disk may not be successful. If the 51% majority consensus rule is not met, the boot
attempt fails, and you must provide the root password to enter single user mode.
Perform the following procedure:
− Provide the root password to enter single user mode.
− Confirm which metadatabase replicas are faulty (metadb -i).
− Typically, if the primary disk were faulty, running metastat would reveal that most
or all submirrors on the primary are in the Maintenance state.
− If deleting the metadatabase replicas from the metadatabase partition of the
primary disk still results in the 51% consensus algorithm being violated, add more
replicas to the metadatabase partition on the secondary disk (metadb -a -c
<number to add> <metadatabase slice name on the secondary disk>).

©
− Delete the metadatabase replicas from the primary disk (metadb -d -f <metadb
slice name of primary disk>). Ignore the mddb.cf error message.
− Verify that the invalid metadatabases were deleted and valid replicas were added
to satisfy the 51% consensus algorithm (metadb -i).
− Reboot the server of the secondary disk to resume operations (sync, halt, and
boot disk1).
− Until a replacement disk is available or the existing primary disk can be repaired,
all reboots should be performed off the secondary disk. (For details, see the
Solaris 8 Administration Guide.)
− Once the replacement disk is available and a maintenance window is available,
boot to single user mode on the secondary disk (stop BroadWorks, sync, halt,
boot <diskn> -s).
− Physically replace the problem disk with the replacement disk, or repair the disk
according to the instructions in the Solaris 8 Administration Guide.
− Repartition the new disk with the same slice information as the secondary disk
(prtvtoc <raw device name of secondary disk> pt, fmthard -s pt <raw device name
of primary disk>).
− Re-add metadatabase replicas to the primary disk (metadb -a -c 3 <primary disk
− Reboot off the secondary disk (sync, halt, boot <diskn>).
− Re-enable all submirrors on the primary disk by issuing the metareplace
command successively for root, swap, and /bw submirrors (for each submirror on
the primary disk, metareplace -e <mirror metadevice name> <faulty slice name>).
This synchronizes the submirrors on the primary disk with the submirrors on the
secondary disk.
− Once re-synchronized, you can, if desired, reboot from the original device (stop
BroadWorks, reboot).
3) If the 51% majority consensus rule is met, the boot attempt succeeds. Perform the
following steps:
− Confirm which metadatabase replicas are faulty (metadb -i).
− Typically, if the primary disk were faulty, then running metastat would reveal that
most or all submirrors on the primary are in the maintenance state.
− Until the replacement disk is available or the existing primary disk can be
repaired, all reboots should be performed off the secondary disk. (For more
information, see the Solaris 8 Administration Guide.)
− Once the replacement disk is available and a maintenance window is available,
boot to single user mode on the secondary disk (stop broadworks, sync, halt, boot
<diskn> -s).
− Delete the failed metadatabase replicas from the primary disk (metadb -d
<primary disk metadb slice name>). Ignore the mddb.cf error message.
− Either physically replace the problem disk with a replacement disk, or repair the
disk according to the instructions in the Solaris 8 Administration Guide.
− Repartition the new disk with the same slice information as the secondary disk
(prtvtoc <raw device name of secondary disk> pt, fmthard -s pt <raw device name
of primary disk>).

©
− Re-add metadatabase replicas to the primary disk (metadb -a -c 3 <primary disk
− Reboot off the secondary disk (sync, halt, boot <diskn>).
− Re-enable all submirrors on the primary disk, by issuing the metareplace
command successively for root, swap, and /bw submirrors (for each submirror on
the primary disk, metareplace -e <mirror metadevice name> <faulty slice name>).
This synchronizes the submirrors on the primary disk with the submirrors on the
secondary disk.
− Once re-synchronized, you can, if desired, reboot from the original device (stop
BroadWorks, reboot).
Following is an example in which the primary disk was completely faulty and resulted in
primary disk boot failure followed by the 51% majority consensus rule not being met when
the boot is attempted off the secondary disk. It uses the same partitioning and mirroring
scheme as documented in the fresh installation section of the BroadWorks Software
Management Guide. Although it may vary from platform to platform, the primary disk (boot
disk) is typically disk0, and the secondary disk is disk1. The example would have to be
adjusted if the partitioning and mirroring scheme is different.
<provide root password to enter single user mode after booting off
secondary disk>
# metadb –i
# metastat
# metadb -d -f c0t0d0s3
<ignore the mddb.cf error message>
# metadb -i
# sync
# halt
ok boot disk1
<stop broadworks>
# sync
# halt
ok boot disk1 –s
<physically replace disk, or repair disk>
# fmthard –s pt /dev/rdsk/c0t0d0s2
# metadb –a –c 3 c0t0d0s3
# sync
# halt
ok boot disk1
# metareplace –e d10 c0t0d0s0
<verify the sync operation has been completed>
<perform following steps if verification of ability to boot off primary
disk is desired>
<stop broadworks>
# reboot

©
19.2.2.2 Insufficient Metadatabase Replicas Due to Transitory Problems
If there are insufficient metadatabase replicas to satisfy the majority consensus algorithm,
(replicas became invalid due to transitory reasons such as power loss, disk I/O glitches,
and so on) and the system is rebooted prior to remedying the situation, the boot sequence
fails. Typically, on the console is a set of messages such as, “Insufficient metadatabase
replicas located. Use metadb to delete …”. The system asks for the root password for
system maintenance.
1) Provide the root password for system maintenance.
2) Determine which metadatabase replicas are faulty (metadb -i). This determines from
which partition to delete metadatabases.
3) If deleting the metadatabase replicas from a metadatabase partition result in the 51%
consensus algorithm being violated, add more replicas to the metadatabase partition
on the other disk (metadb -a -c <number to add> <metadatabase slice name on the
other disk>).
4) Delete the metadatabase replicas from the affected disk (metadb -d -f <metadb slice
name of affected disk>). Ignore the mddb.cf error message.
5) Verify that the invalid metadatabases were deleted and valid replicas were added to
satisfy the 51% consensus algorithm (metadb -i).
6) Reboot the server to resume operations (reboot).
7) Re-add metadatabase replicas to the affected disk (metadb -a -c 3 <metadb slice
name of affected disk>).
8) Verify the above replicas were re-added (metadb -i).
documented in the fresh installation section of the BroadWorks Software
Management Guide. In this example, assume that metadatabase replica(s) on the
primary disk are invalidated, and that two more metadatabase replicas must be added
to the secondary disk to satisfy the 51% consensus algorithm. The example has to be
adjusted if the partitioning and mirroring scheme is different.
<enter root password for system maintenance>
# metadb -i
<ignore the mddb.cf error message resulting from the above command>
# metadb -I
# reboot
# metadb -i

©
19.2.2.3 Insufficient Metadatabase Replicas Due to Faulty Secondary Disk
If the secondary disk was completely faulty but it was not replaced, and a reboot was
performed, then the boot sequence fails. Typically, on the console is a set of messages
such as, “Insufficient metadatabase replicas located. Use metadb to delete …”. The
system asks for the root password for system maintenance.
1) Provide the root password for system maintenance.
2) Determine which metadatabase replicas are faulty (metadb -i).
3) If deleting the metadatabase replicas from the metadatabase partition of the
secondary disk results in the 51% consensus algorithm being violated, add more
replicas to the metadatabase partition on the primary disk (metadb -a -c <number to
add> <metadatabase slice name on the primary disk>).
4) Delete the metadatabase replicas from the secondary disk (metadb -d -f <metadb
slice name of affected disk>). Ignore the mddb.cf error message.
5) Verify that the invalid metadatabases were deleted and valid replicas were added to
satisfy the 51% consensus algorithm (metadb -i).
6) Reboot the server to resume operations (reboot).
7) Detach all submirrors on the problem disk by issuing the metadetach commands
disk, metadetach -f <mirror> <submirror>).
8) Once the replacement disk is available and a maintenance window is available, boot
to single user mode (stop broadworks, sync, halt, boot -s).
9) Either physically replace the problem disk with the replacement disk or repair the disk
according to the instructions in the Solaris 8 Administration Guide.
10) Repartition the new disk with the same slice information as the primary disk (prtvtoc
<raw device name of primary disk> pt, fmthard -s pt <raw device name of secondary
disk>).
11) Re-add state databases to the secondary disk (metadb -a -c 3 <metadb slice name on
secondary disk>).
12) Verify the above replicas were re-added (metadb -i).
13) Reboot.
14) Re-attach all submirrors on the replaced disk, by issuing the metattach command
disk, metattach <mirror> <submirror>). This causes the submirrors on the secondary
disk to re-synchronize.
documented in the fresh installation section of the BroadWorks Software
Management Guide. The example has to be adjusted if the partitioning and mirroring
scheme is different.
<enter root password for system maintenance>
# metadb -i
<ignore the mddb.cf error message resulting from the above command>
# metadb -i
# reboot

©
<stop broadworks>
# sync
# halt
# boot -s
<physically replace secondary disk or repair secondary disk>
# metadb -i
# reboot
# metattach d10 d30
# metattach d11 d31
# metattach d14 d34

©
20 BroadWorks System Monitoring
Proper BroadWorks system monitoring is comprised of three distinct tasks:

 Platform Resource Monitoring – Proactively monitoring key platform resource
indicators to make sure they are operating within acceptable ranges.
 Performance Measurement Monitoring – Proactively polling key SNMP gauges
and counters to gather information on a server’s health.
 SNMP Trap Monitoring – Reactively responding to traps generated by the
BroadWorks servers.
This section outlines the system monitoring that must be performed to make sure proper
BroadWorks operations.
20.1 Platform Resource Monitoring – Key Resource Indicators

The health of BroadWorks relies on the health of the servers that it runs on. The servers
must be monitored to make sure they are operating correctly. The following key resource
indicators should be monitored:
 File system (disk usage)
 CPU usage
 Memory usage/Paging
 Disk activity
 Execution Server heap usage (Application Server and Network Server)
 Database size (Application Server and Network Server)
As of Release 14.sp1, the BroadWorks Element Management System (EMS)
automatically collects key resource indicators for all BroadWorks nodes.
NOTE: All servers should be running with key resource indicators that are within acceptable
ranges. Acceptable ranges for each of the key resource indicators listed above are defined in
the BroadWorks System Engineering Guide.
20.1.1 File System Monitoring

It is critical that the server file system does not reach 100% utilization; otherwise, the
application ceases to function. In general, disk partition should be running under 75%
utilization.
20.1.1.1 Collecting File System Indicator

File system use can be obtained through the following means.
20.1.1.1.1 Command Line

File system usage can be viewed manually from the UNIX/LINUX prompt using the df –k
command. The Capacity field indicates the percentage of partition in use.
IHApp$ df –k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 1987399 1208093 719685 63% /
/proc 0 0 0 0% /proc

©
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1395576 16 1395560 1% /var/run
swap 1395616 56 1395560 1% /tmp
/dev/md/dsk/d13 13376379 6977170 6265446 53% /bw
20.1.1.1.2 SNMP
On Solaris systems, file system usage can also be monitored through SNMP by polling
the Sun Management Center (SMC) mib file register krUFSFileSystemGroup.
On Linux systems, file system usage can also be monitored through SNMP by polling the
UCD-SNMP-MIB.txt MIB file register dskPercent.
20.1.1.2 Healthmon File System Check

The healthmon script automatically monitors the /, /bw, and /tmp partitions (if present),
every time it runs and generates the following traps severities:
 Low severity trap at 80% occupancy on any partition
 High severity trap at 90% occupancy on any partition
 Critical severity trap at 95% occupancy on any partition
 Critical severity trap and a stopbw at 98% occupancy on any partition
The healthmon script stops the BroadWorks application at 98% partition occupancy to
avoid possible database corruption.
20.1.2 CPU Usage Monitoring

System CPU usage must be monitored to make sure proper BroadWorks operation. High
CPU usage bottlenecks the application and can lead to outages.
CPU usage is generally broken down into the following run time percentages:
 User – The percentage of time the CPU is spending on user processes, such as
applications, shell scripts, or interacting with the user
 System – The percentage of time the CPU is spending executing kernel tasks
 I/O Wait – The percentage of time the CPU is waiting for input or output from a block
device, such as a disk
 Idle – The percentage of time the CPU is not doing anything useful
For both Solaris and Linux systems, CPU busy occupancy is defined as the sum of CPU
User time and CPU System time. I/O Wait time, although possibly indicative of an I/O
issue, does not contribute to CPU busy and it should be considered as “Idle” time. Certain
monitoring utilities (for example, vmstat on Solaris) already consider this and include I/O
Wait in the “Idle” value.
20.1.2.1 Collecting CPU Usage Indicator

Busy CPU percentage (User Time + System Time) can be collected through various
means. It is recommended that averaged CPU usage be gathered as opposed to
instantaneous CPU usage values. BroadSoft recommends a five-minute averaging
window.

©
CPU usage can also be monitored through SNMP on Solaris by polling the Sun
Management Center (SMC) mib file register krCPUUtilGroup, and on Linux by polling
UCD-SNMP-MIB.txt, but this is not the recommended approach since it reports an
instantaneous CPU usage ready that may be misleading.
The recommended methodology for tracking CPU usage is through either System Activity
Reporter (SAR) or vmstat utilities with a five-minute average.
20.1.2.1.1 System Activity Reporter

The Solaris System Activity Reporter (SAR) tool provides historical information on system
activity such as CPU usage, paging, and swap space usage. SAR should be activated on
all servers with a five-minute collection window. This provides an excellent mechanism to
retrieve CPU averages for servers. For information on SAR activation, consult the
BroadWorks Software Management Guide.
To view CPU average usage activity, from the UNIX/LINUX prompt type sar.
Example:
as1$ sar
SunOS IHApp 5.8 Generic_108528-19 sun4u 08/08/03
00:00:01 %usr %sys %wio %idle

00:05:00 25 5 12 57
00:10:00 27 5 15 53
00:15:01 22 4 11 62
00:20:00 27 6 12 55
……..
11:45:00 28 6 10 56
11:50:00 34 7 9 50
11:55:01 35 7 10 48
12:00:01 29 7 10 54
12:05:01 29 7 17 47
Average 27 5 14 53
Total CPU Busy percentage for any given five-minute interval would be equal to %usr +
%sys.
Historical SAR results are stored in /var/adm/sa on Solaris systems and /var/log/sa on
Linux systems. Historic files can be viewed using the sar –f <filename> option.
20.1.2.1.2 vmstat
The vmstat utility can be used to manually gather averaged CPU information.
To collect five-minute averaged CPU usage, from the UNIX/LINUX prompt type vmstat
300 <count>. The <count> parameter would be the number collections to perform.
Example:
as1$ vmstat 300 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id
0 0 0 3351880 162248 30 160 23 3 4 0 1 1 0 5 1 420 506 738 3 2 95
0 0 0 3489880 255752 51 322 0 1 1 0 0 1 0 3 1 411 2242 1454 3 2 95
0 0 0 3490504 255328 31 197 0 1 1 0 0 1 0 3 1 367 1845 1392 2 1 97
0 0 0 3494720 257776 15 88 0 1 1 0 0 1 0 3 1 377 1584 1408 2 1 97
0 0 0 3492504 257352 66 436 1 2 2 0 0 2 0 4 2 395 2637 1478 5 2 92

©
Average CPU Busy percentage for each five-minute interval would be equal to us + sy.
The first reading returned by vmstat should to be discarded.
20.1.2.2 bwCPUMon script CPU Monitoring

The bwCPUMon scheduled task uses SAR the monitor average CPU usage. The
bwCPUMon script should be enabled to run every six minutes and generates the following
severity traps when the SAR five-minute %idle average surpasses the following
thresholds:
 MINUMUM_CPU_IDLE_TIME_MEDIUM=30
 MINUMUM_CPU_IDLE_TIME_HIGH=20
 MINUMUM_CPU_IDLE_TIME_CRITICAL=10
20.1.3 Memory Usage/Paging Monitoring

A lack of platform memory can slow the BroadWorks application down and increases CPU
usage. A server that has exhausted physical memory starts swapping. Swapping refers
to moving processes in and out of physical memory when the system is extremely short of
physical memory needed for use by the processes. Swapping places a heavy load on
CPU, disk I/0 subsystems, and Java. Ideally, the BroadWorks server should not be
swapping.
The easiest way to monitor memory swapping is to monitor server paging activity. Ideally,
there should be no paging activity.
20.1.3.1 Collecting Paging Indicator

The SAR utility provides an indication of paging activity on both Solaris and Linux systems.
Solaris and Linux manage memory differently and therefore report paging activity in
different fashions.
20.1.3.1.1 Solaris – SAR Paging

To view five-minute averaged paging activity data, from the UNIX prompt enter sar -g.
The pgscan/s parameter provides an indication of average paging activity, which ideally
should be zero.
Example:
bwadmin@IHApp$ sar -g
SunOS IHApp 5.9 Generic_117171-12 sun4u 07/29/2005

00:00:01 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
00:05:00 0.16 0.28 0.27 0.00 0.00
00:10:00 0.03 0.03 0.03 0.00 0.00
00:15:00 0.03 0.03 0.03 0.00 0.00
00:20:00 0.04 0.04 0.04 0.00 0.00
00:25:00 0.03 0.03 0.03 0.00 0.00
00:30:01 0.03 0.03 0.03 0.00 0.00
00:35:00 0.05 0.05 0.05 0.00 0.00
00:40:00 0.03 0.03 0.03 0.00 0.00
00:45:01 0.03 0.04 0.04 0.00 0.00
00:50:00 0.04 0.05 0.04 0.00 0.00

©
20.1.3.1.2 Linux – SAR Paging
To view five-minute averaged paging activity data, from the LINUX prompt enter sar -W.
The pswpin/s parameter provides an indication of average paging activity, which ideally
should be zero.
Example:
bwadmin@serv1$ sar -W
Linux 2.6.9-42.0.2.ELsmp (serv1) 01/31/2007
12:00:01 AM pswpin/s pswpout/s

12:05:01 AM 0.00 0.00
12:10:01 AM 0.00 0.00
12:15:01 AM 0.00 0.00
12:20:01 AM 0.00 0.00
12:25:01 AM 0.00 0.00
12:30:01 AM 0.00 0.00
20.1.3.2 bwCPUMon Script Paging Monitoring

The bwCPUMon scheduled task uses SAR to monitor average paging activity. On Solaris
systems, the script generates a Critical severity trap when the SAR five-minute pgscan/s
reading is over 200. On Linux systems, the script generates a Critical severity trap when
the SAR five-minute pswpin/s average is two or greater for last three five-minute
windows.
20.1.4 Disk Activity

Although BroadWorks applications tend not to be I/O blocking, excessive disk activity can
be an indication of too much I/O traffic, such as writing to disk. Excessive writing to disk
can be caused by a number of correctable activities such as logging levels set too high for
a server. It is important to monitor disk activity to make sure that disks are not
bottlenecked.
20.1.4.1 Collecting Disk Activity Indicator

The iostat utility provides an indication of I/O activity on both Solaris and Linux systems.
20.1.4.1.1 Solaris – iostat

To view five-minute averaged disk activity data, from the UNIX prompt enter iostat –cx
300 2. The svc_t (average service time of the disk) and %b (percentage of time the disk
is busy) parameter provides an indication of disk activity. The first reading should be
discarded.
Example:
bwadmin@as1$ iostat -x 300 2
device r/s w/s kr/s kw/s wait actv svc_t %w %b

md10 0.0 0.6 0.3 10.6 0.0 0.0 37.5 0 1
md11 0.0 0.0 0.0 0.0 0.0 0.0 15.1 0 0
md14 9.8 56.2 193.8 1652.7 0.0 1.3 20.4 3 39
md20 0.0 0.6 0.1 10.6 0.0 0.0 26.6 0 0
md21 0.0 0.0 0.0 0.0 0.0 0.0 10.5 0 0
md24 4.9 56.2 97.0 1652.7 0.0 1.0 16.0 0 35
md3 0.0 0.6 0.1 10.6 0.0 0.0 26.5 0 0

©
20.1.4.1.2 Linux – iostat
To view five-minute averaged disk activity data, from the LINUX prompt enter iostat –
dx 300 2. The svctm (average service time of the disk) and %util (percentage of time
the disk is busy) parameters provides an indication of disk activity. The first reading
should be discarded.
Example:
bwadmin@serv1$ iostat -dx 300 2
Linux 2.6.9-42.0.2.ELsmp (serv1) 02/01/2007
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s

avgrq-sz avgqu-sz await svctm %util
sda 0.00 14.23 0.01 3.64 0.44 142.94 0.22 71.47
39.29 0.02 4.63 0.52 0.19
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
57.70 0.00 3.42 3.42 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s

avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.28 0.00 0.31 0.00 4.67 0.00 2.33
15.22 0.00 1.01 0.66 0.02
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
20.1.5 Thread Activity

Whenever the server processors are overstressed, the run queue can exceed the number
of CPUs, which can lead to performance bottlenecks. It is important to monitor thread
activity to make sure that processes are not bottlenecked. The number of threads waiting
in the run queue versus available CPUs can be measured to see if bottlenecks are
occurring.
20.1.5.1 Collecting Thread Activity Indicator

To view five-minute averaged thread activity, from the UNIX or LINUX prompt enter
vmstat 300 2. The r (number of threads waiting in the run queue) and b (number of
threads blocked due to I/O blocking) parameters provide an indication of thread activity.
The first reading should be discarded.
Example:
as1$ vmstat 300 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id
0 0 0 3351880 162248 30 160 23 3 4 0 1 1 0 5 1 420 506 738 3 2 95
0 0 0 3489880 255752 51 322 0 1 1 0 0 1 0 3 1 411 2242 1454 3 2 95
0 0 0 3490504 255328 31 197 0 1 1 0 0 1 0 3 1 367 1845 1392 2 1 97
0 0 0 3494720 257776 15 88 0 1 1 0 0 1 0 3 1 377 1584 1408 2 1 97
0 0 0 3492504 257352 66 436 1 2 2 0 0 2 0 4 2 395 2637 1478 5 2 92
The number of CPUs on Solaris servers can be obtained using the uname –X command.
The number of CPUs on Linux servers can be obtained using the cat /proc/cpuinfo
command.

©
20.1.6 Application Server Execution Server Heap Monitoring
The Application Server Execution Server (XS) process is assigned a fixed amount of heap
memory at startup. The heap usage grows over time until Java cleans up the heap with a
full garbage collection. The Application Server Execution Server (XS) heap is cleaned up
whenever the heap reaches 68% of maximum allocated heap.
The heap size after the full collection represents the actual heap required to support the
users and active calls at the time. It is important that the post clean up heap does not
grow to 68% of the maximum allocated heap or the heap ends up in a full collection loop,
which severely affects server performance.
The post Java garbage collection heap size needs to be monitored daily and should
always be <= 60% of the allocated heap. As the heap increases towards 60%, it may be
possible to increase the maximum allocated heap by adding more memory to the server.
20.1.6.1 Collecting Heap Size Indicator

The following information should be collected from the primary Application Server since it
should be handling all traffic. Heap garbage collection (GC) logging information is stored
in /var/broadworks/logs/appserver/XSOutputXX.log files. The GC line AFTER any line
containing “CMS-concurrent-reset:” indicates the heap size after a full garbage collection.
For example, the GC line after CMS-concurrent-reset …
225328.399: [GC 225328.399: [ParNew: 24448K->0K(24512K), 0.1030736 secs]
853962K->834988K(1572800K), 0.1033765 secs]
225328.707: [CMS-concurrent-sweep: 6.553/7.059 secs]
225328.707: [CMS-concurrent-reset-start]
225328.914: [CMS-concurrent-reset: 0.207/0.207 secs]
815126K->796196K(1572800K), 0.0888862 secs]
… indicates that 796196K is the heap size after a full GC collection and 1572800K is the
maximum heap size. After this full collection, 796196/1572800 or 51% of the 1.5 GB heap
is in use.
To obtain the maximum post-collection heap size for a given day, the XSOutputXX.log
files should be analyzed. The post full collection heap sizes throughout the day can be
obtained using a simple awk statement such as:
$ cat XSOutput24.log |awk '$2=="[CMS-concurrent-reset:"{getline;print}' >> post-
coll-heap.out
You can now obtain the maximum post-collection heap size from the post-coll-heap.out
file. For example, if post-coll-heap.out contains the following.
bwadmin@serv1$ sar –W
608826K->588221K(1572800K), 0.0693588 secs]
606385K->585812K(1572800K), 0.0711760 secs]
608213K->587808K(1572800K), 0.0679470 secs]
606974K->586070K(1572800K), 0.0834866 secs]
606804K->586216K(1572800K), 0.2496168 secs]
610465K->589446K(1572800K), 0.0682067 secs]

©
599254K->578091K(1572800K), 0.0613286 secs]
594655K->574355K(1572800K), 0.0703674 secs]
… 589446K is the maximum post-collection heap size for that day and 1572800K is the
configured maximum heap size.
20.1.6.2 bwCPUMon Script Heap Usage Monitoring

The cpuMon scheduled task monitors overall Execution Server heap usage to make sure
the heap is not growing out of control. The cpuMon task generates a Critical severity trap
when the percentage of heap used goes above 85%. If you receive
bwMemoryLimitReached traps, contact BroadSoft Support, as this may be an indication
of a memory leak or a heap size that is too small.
20.1.7 Database Size Monitoring

The Application Server and Network Server TimesTen database is assigned a fixed
permanent size allocation at installation. The database grows as the system scales. It is
important that the database permanent size in use does not grow beyond 90% of the
maximum allocated permanent size. As the database permanent size increases toward
90%, it may be possible to increase the maximum allocated permanent size depending on
platform memory.
NOTE: Database size monitoring for the Database Server (Oracle database) is done through
File System Monitoring.
20.1.7.1 Collecting Database Size Indicator

On both the Application Server and the Network Server, the database allocated
permanent size and permanent size in use can be obtained using the TimesTen ttIsql
utility dssize command.
Example:
bwadmin@IHApp$ ttIsql
Command> dssize;
PERM_ALLOCATED_SIZE: 141312
PERM_IN_USE_SIZE: 69966
PERM_IN_USE_HIGH_WATER: 70412
TEMP_ALLOCATED_SIZE: 43008
TEMP_IN_USE_SIZE: 12693
TEMP_IN_USE_HIGH_WATER: 16276
This database has 69966/141312=49% of permanent size in use.
20.1.7.2 healthmon Script Database Size Monitoring

The healthmon scheduled task monitors database permanent size usage to make sure
the system provider receives an indication when the database size passes the 80%
threshold. If possible, the database should be resized once that threshold is reached.

©
20.2 BroadWorks Performance Measurements Monitoring
BroadWorks provides a number of SNMP counters and gauges that can be polled to
provide useful information. All performance measurements are accessible via the CLI or
through external polling. Counters can be cleared via the SNMP set command.
It is important to note that in certain cases, excessive polling can affect server
performance. For example, one should NOT poll all Application Server performance
measurements every minute. Different MIBs should be polled at different intervals to best
balance granularity, requirements, and performance. In addition, polling intervals should
be set so that they do NOT run exactly on the hour, half hour, or quarter hour to avoid
interfering with internal application scheduled tasks.
For more information on how to configure SNMP and for a complete list of BroadWorks
SNMP MIBs, see the BroadWorks Performance Measurements Interface Specification
Guide.
The following sections describe best practices for monitoring performance measurements.
Similar to platform resources, there are performance measurements that are considered
key performance indicators. Key performance indicators are identified. A minimum
number of providers should be collecting all key performance indicators. A summary of all
key performance indicators is also provided in the BroadWorks System Engineering
Guide.
A table is provided at the end of each section outlining what should be monitored. The
table provides the suggested interval (Int) frequency [daily (D), hourly (H), 30 minutes (T),
15 minutes (F)], and the type (Type) of measurement. Trending data (T) is not indicative
of a problem, but rather valuable information from a tracking perspective. Performance
data (P) is information that can be collected and measured or compared against defined
thresholds, indicating a server may be encountering performance issues.
20.2.1 Media Server

The Media Server provides a number of SNMP counters and gauges that can be used to
monitor performance. These MIBs can be polled from an external management system or
via the CLI.
20.2.1.1 Media Server Usage

The current number of Media Server ports in use can be monitored (msPortsInUse) as
well as any port allocation failures (msNoPortAvailableErrors) due to licensing violations.
The total number of interactive voice response (IVR) and three-way conferencing sessions
since the last clearing of the counters is provided by the msIvrSessionCount and
msConferenceCount counters.
Example:
MS_CLI/Monitoring/PM/MediaServer> get mcp/msPortsInUse
msPortsInUse 16
MS_CLI/Monitoring/PM/MediaServer> get mcp/msNoPortAvailableErrors

*msNoPortAvailableErrors 2
MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrSessionCount

*msIvrSessionCount 199170
MS_CLI/Monitoring/PM/MediaServer> get mcp/conferencing/msConferenceCount

*msConferenceCount 0

©
msNoPortAvailableErrors – Every instance of msNoPortAvailableErrors means that the
Media Server refused a session due to licensing. The Application Server routes to the
next Media Server in the Media Server list (if one is provided).
Note that:
 Ideally, msNoPortAvailableErrors should be zero.
 If you observe instances of msNoPortAvailableErrors on all Media Servers, it probably
means that calls requiring Media Server resources are being dropped.
 Constant high values of msNoPortAvailableErrors on all Media Servers generally
means that additional Media Servers are required to meet network traffic
requirements.
 If you observe instances of msNoPortAvailableErrors on one Media Server, it
probably means that the Media Server Selection (MSS) policy on the Network Server
is returning one Media Server disproportionately. In this case, the MSS algorithm
should be revisited.
msPortsInUse (Key Performance Indicator) – This gauge can be used to monitor actual
Media Server port usage and is also an excellent way to monitor Media Server system
capacity (that is, it can be used to determine if additional Media Servers are required in the
network). It can also be used to determine busy hour Media Server bottlenecks. The
msPortsInUse gauge is a weighted representation of port usage. For example, a G.729
call would result in 1.7 ports in use. msPortsInUse is rounded up after all ports in use are
calculated.
msIvrSessionCount and msConferenceCount – These counters provide a view of a
Media Server session over time and they can be used to plot daily Media Server usage.
You can poll these once an hour to derive Media Server busy-hours and help better
understand overall Media Server usage.
Performance Int Node Type Threshold
Measurement
msNoPortAvailableErrors H MS P Ideally zero, unless Media Server blocking

is acceptable.
The blocking percentage equals
[msNoPortAvailableErrors /
(msIvrSessionCount +
msConferenceCount)] * 100.
msPortsInUse F MS P Yes, low severity at 80%, medium severity

at 90% of total port usage.
msIvrSessionCount H MS T No, trending data only.
msConferenceCount H MS T No, trending data only.
20.2.1.2 Voice Messaging SMTP Send

Voice messaging recordings are relayed to an SMTP server for subsequent delivery to a
mail server. You can monitor voice mail send problems by reviewing the
msPrimarySmtpErrors and msSecondarySmtpErrors counters. Note that an SMTP error
results in a SNMP trap as well.
Example:
MS_CLI/Monitoring/PM/MediaServer> get smtp/primary/msPrimaryEmailSent
*msPrimaryEmailSent 32206

©
MS_CLI/Monitoring/PM/MediaServer> get smtp/primary/msPrimarySmtpErrors
*msPrimarySmtpErrors 0
MS_CLI/Monitoring/PM/MediaServer> get smtp/secondary/msSecondaryEmailSent

*msSecondaryEmailSent 0
MS_CLI/Monitoring/PM/MediaServer> get
smtp/secondary/msSecondarySmtpErrors
*msSecondarySmtpErrors 0
msPrimaryEmailSent and msSecondaryEmailSent – These counters provide an

historic view of SMTP voice mail forwarding. Monitoring these counters provides a view of
SMTP traffic that is related to Media Servers.
msPrimarySmtpErrors and msSecondarySmtpErrors – These counters are pegged
whenever the Media Server times out or receives a server error from the SMTP server.
Ideally, these counters should be zero.
If the primary SMTP server does not respond or returns a server error, the Media Server
tries the secondary SMTP server and pegs the msPrimarySmtpErrors counter. If the
secondary SMTP server does not respond or returns an error, msSecondarySmtpErrors is
pegged and the recorded voice mail is lost. The Media Server does not perform any local
queuing, so it is critical that there is always an available SMTP server.
The Media Server SMTP timeout is controlled by the MS_CLI/System> parameter
smtpTimeout and it is set to “30 seconds” by default. You may want to consider
increasing this if you are experiencing problems.
SMTP servers are passed to the Media Server by the Application Server and are under
AS_CLI/Interface/Mail>. For redundancy purposes, both primarySMTPServer and
secondarySMTPServer should be defined.
Measurement
msPrimaryEmailSent H MS T No, trending data only.
msPrimaryEmailSent H MS T No, trending data only.
msPrimarySmtpErrors H MS T No, trending data only.
msPrimarySmtpErrors H MS T No, trending data only
20.2.1.3 IVR Memory

The Media Server requires memory for media playing and recording. The IVR memory
maximum is statically defined under MS_CLI/Service/IVR> parameter memorySize. If the
Media Server cannot allocate memory, the session fails. The SNMP trap
msMemAllocFailure is generated each time the Media Server terminates a session due to
lack of IVR memory.
20.2.1.3.1 Real-Time Memory Usage

Real-time memory usage can be monitored using the msIvrAudioMemoryInUse and
msIvrFreeAudioMemory gauges.
MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrFreeAudioMemory

msIvrFreeAudioMemory 500705896

©
MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrAudioMemoryInUse
msIvrAudioMemoryInUse 8467648
msIvrAudioMemoryInUse and msIvrFreeAudioMemory – These gauges can be used

to monitor actual Media Server IVR memory usage for proactive memory monitoring. At
any given time, they provide a snapshot of memory usage. As such, the readings
provided can vary dramatically depending on the polling interval. The shorter the interval,
the more accurate the memory usage information is, although it is not a good idea to poll
too frequently. For example, every minute would probably be too low a granularity, while
every 15 minutes would be good enough for the purposes of these MIBs.
Note that:
 From a more proactive perspective, receiving any msMemAllocFailure trap is an
indication that IVR memory should be increased.
 If active msIvrAudioMemoryInUse/(msIvrAudioMemoryInUse +
msIvrFreeAudioMemory) average memory in use, during busy hour, is more than
75%, it would be best to increase the IVR memory.
 Default memorySize is set to 120 MB. Generally, 50% of the available physical
memory should be allocated to the IVR process at initial installation. This allows the
server to better handle expected traffic without having to tweak memorySize.
 The maximum amount of IVR memory that can be allocated on a server is 2 GB.
 Resizing the MS_CLI/Service/IVR parameter memorySize requires a Media Server
stop and start.
Measurement
msIvrAudioMemoryInUse F MS P Yes. If average memory in use

/(msIvrAudioMemoryInUse during busy hour is more than 75%
+ msIvrFreeAudioMemory) (low alarm), or more than 85%
(high alarm).
20.2.1.4 RTP Statistics

The Media Server reports Real-Time Transport Protocol (RTP) system-wide statistics.
These statistics should be viewed as informational only and not be taken as a definitive
indication that there is a quality of service (QoS) problem.
20.2.1.4.1 Receive RTP

The Media Server provides a number of performance measurements related to RTP
received by the Media Server.
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpPacketsExpected
msRtpPacketsExpected 197273595
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpPacketsReceived

msRtpPacketsReceived 196818032
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpOutOfOrder

msRtpOutOfOrder 203444
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpBadPayload

msRtpBadPayload 11669
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpSsrc

©
msRtpSsrc 13004
msRtpPacketsReceived/msRtpPacketsExpected – These counters provide a general

view on received packet loss. Note that since msRtpPacketsExpected is reported by the
remote party via Real-Time Control Protocol (RTCP), it is only as accurate as the device
sending it and should be used as only one indicator of a potential problem.
The msRtpPacketsReceived/msRtpPacketsExpected ratio can be used to help identify
QoS issues related to the Media Server. The acceptable ratio depends on the network
and codec selection, but in general, a receive loss of less than 2.5% is considered
acceptable.
A good practice would be to start monitoring the received loss ratio once a day and if high
results are obtained, reduce the granularity to identify if the overall result is evenly
distributed or a result of loss spikes.
msRtpOutOfOrder – This counter provides an indication of how many RTP packets are
received out of sequence. A high percentage of out-of-order packets versus
msRtpPacketsReceived may be an indication of a routing problem. However, the Media
Server automatically reorders the packets so there should be no impact on performance.
msRtpBadPayload – This counter reports the number of RTP packets received with an
unknown payload (not 0 for 711, 2 for 726, 18 for 729, or rfc2833 dynamic payload).
msRtpSsrc – This counter reports the number of packets received with a bad
synchronization source (ssrc). The packet ssrc must be the same for all RTP packets in a
stream. A high number of any of these with respect to msRtpPacketsReceived could
indicate an issue with the far-end device.
msRtpReceivedPacketJitter – Disregard this counter as it provides no value.
20.2.1.4.2 Transmit RTP

The Media Server provides a number of performance measurements related to RTP
transmitted by the Media Server.
rtp/transmit/msRtpCumulativePacketsLost
msRtpCumulativePacketsLost 2738437
MS_CLI/Monitoring/PM/MediaServer> get rtp/transmit/msRtpPacketsSent

msRtpPacketsSent 195136008
MS_CLI/Monitoring/PM/MediaServer> get rtp/transmit/msRtpFramesSkipped

msRtpFramesSkipped 11079
msRtpCumulativePacketsLost/msRtpPacketsSent – These counters provide a general

view on transmit packet loss. Note that since msRtpCumulativePacketsLost is reported by
the remote party via RTCP, it is only as accurate as the device sending it and should be
used as only one indicator of a problem.
The msRtpCumulativePacketsLost/msRtpPacketsSent ratio can be used to help identify
QoS issues related to the Media Server. The acceptable ratio depends on the network
and codec selection, but in general, a transmit loss of less than 2.5% is considered
acceptable.
A good practice would be to start monitoring the received loss ratio once a day and if high
results are obtained, reduce the granularity to identify if the overall result is evenly
distributed or a result of loss spikes.

©
msRtpFramesSkipped – This counter indicates how many frames were dropped by the
Media Server on the transmit side due to a resource issue (for example, CPU). A value of
less than a five-unit increase over a ten-second period is considered acceptable. A higher
value than that affects voice quality.
msRtpFramesSkipped should be monitored periodically. A good practice would be to poll
the delta in msRtpFramesSkipped every 15 minutes and divide by 90. If the value is more
than five, investigation is required.
Performance Measurement Int Node Type Threshold
msRtpCumulativePacketsLost H MS P Yes. The acceptable ratio

/msRtpPacketsSent depends on the network and
codec selection, but in general, a
transmit loss of less than 2.5% is
considered acceptable.
msRtpPacketsReceived/msRt H MS P Yes. The acceptable ratio

pPacketsExpected depends on the network and
codec selection, but in general, a
transmit loss of less than 2.5% is
considered acceptable.
msRtpFramesSkipped F MS P Yes. Represents RTP frames not

sent due to resource issues. Poll
msRtpFramesSkipped every 15
minutes and divide by 90. If the
result (over four polling intervals) is
more than five (low alarm), or if the
result is more than 10 (high alarm).
20.2.1.5 SIP Statistics

The Media Server uses SIP as the call control protocol. SIP protocol statistics can be
used to identify signaling issues.
MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInviteIns
*msSipStatsInviteIns 265162
sip/msSipStatsInvite200OKRetransmitsOuts
*msSipStatsInvite200OKRetransmitsOuts 1027
MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInviteIns

*msSipStatsInviteIns 265249
MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsByeIns

*msSipStatsByeIns 262823
MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInfoIns

*msSipStatsInfoIns 668622
sip/msSipStatsRequestRetransmittedIns
*msSipStatsRequestRetransmittedIns 43622

©
msSipStatsInvite200OKRetransmitsOuts/msSipStatsInviteIns – These counters
provide an indication of what percentage of 200 OK responses required re-transmission.
By default (under CLI level int/sip>) the Media Server re-transmits the 200 OK if it does not
receive a response within 200 milliseconds. Even with a quick re-transmission timer, the
200 OK re-transmission ratio should be under 5%.
msSipStatsRequestRetransmittedIns/(msSipStatsInviteIns+msSipStatsByeIns +
msSipStatsInfoIns) – These counters provide an indication of re-transmissions requests
from an Application Server. The Application Server request retry ratio should be under
5%.

Measurement
msSipStatsInvite200OKRetr H MS P Yes. These counters provide an

ansmitsOuts/msSipStatsInvit indication of re-transmissions of
eIns 200 OK responses from the
Media Server. The ratio should
be less than 5%.
msSipStatsRequestRetrans H MS P Yes. These counters provide an

mittedIns/(msSipStatsInviteI indication of re-transmissions
ns + requests from the Application
msSipStatsByeIns+msSipSt Server. The Application Server
atsInfoIns) request retry ratio should be less
than 5%.
20.2.1.6 MSCML Statistics

Media Server Control Markup Language (MSCML) is the protocol extension to SIP used
to exchange Media Server directives such as play file and record between an Application
Server and Media Server. There are three commands:
 PlayCollect – Play a .wav file and collect digits. Used for IVR prompt functions such
as voice messaging.
 Play – Play a .wav file without any digit collection. Used to play treatments or
announcements.
 PlayRecord – Play a .wav file (for example, a user’s greeting) and record an incoming
voice message.
The Media Server provides MSCML statistics that can be monitored.
MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLPlayCollectFailed
*msMSCMLPlayCollectFailed 144
MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLPlayFailed

*msMSCMLPlayFailed 3
MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLRecordFailed

*msMSCMLRecordFailed 297
MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLSendMailFailed

*msMSCMLSendMailFailed 10695
msMSCMLPlayFailed, msMSCMLRecordFailed, and msMSCMLPlayCollectFailed –

These counters are pegged in response to a failure condition with respect to the Play,
PlayCollect, or PlayRecord commands. In general, each time these are pegged it means
that a file could not be played and the session with the Media Server has failed.

©
Note that:
 Ideally, all these should be zero.
 These failures are all related to .wav file retrieval, parsing, or playout and there is a
corresponding trap indicating the problem (for example, HTTP get/parsing error).
msMSCMLSendMailFailed – This counter is pegged each time a Media Server receives
an error from an SMTP/POP server that an e-mail could not be delivered. It means that
the SMTP server was reachable, but the Media Server received an error code back.
This error occurs in cases where the SMTP server attempts to forward an e-mail to the
mail server and returns an error code back to the Media Server. A common example
would be an invalid e-mail address assigned to the user’s unified voice messaging
account or an invalid e-mail address on a carbon copy or notification request. There is a
corresponding trap indicating the problem.
Measurement
msMSCMLPlayFailed , D MS T No, trending data only since a

msMSCMLRecordFailed, trap was generated.
msMSCMLPlayCollectFailed
msMSCMLSendMailFailed D MS T No, trending data only since a

trap was generated.
20.2.2 Network Server

The Network Server provides a number of SNMP counters and gauges that can be used
to monitor performance. These MIBs can be polled from an external management system
or via the CLI.
20.2.2.1 Internal Queue Statistics (Key Performance Indicator)

In general, SIP-signaling delays provide a view on any delays that are affecting call
processing. The Network Server provides Execution Server process internal queue
statistics as well.
NS_CLI/Monitoring/PM/NetworkServer> pwd
broadworks/nsExecutionServer/system/internalStats/
NS_CLI/Monitoring/PM/NetworkServer> get bwNSSystemInternalQueueTable

bwNSSystemInternalQueueTable:
(1) bwNSSystemInternalQueueIndex
(2) bwNSSystemInternalQueueName
(3) bwNSSystemInternalQueueSize
(4) bwNSSystemInternalQueueTimeAvg
(5) bwNSSystemInternalQueueTimeMin
(6) bwNSSystemInternalQueueTimeMax
(7) bwNSSystemInternalQueueTimeMaxTimestampMSB
(8) bwNSSystemInternalQueueLengthCurrent
(9) bwNSSystemInternalQueueLengthAvg
(10) bwNSSystemInternalQueueLengthMax
(11) bwNSSystemInternalQueueLengthMaxTimestampMSB
(12) bwNSSystemInternalQueueLengthMaxTimestampLSB
(13) bwNSSystemInternalQueueTimeMaxTimestampLSB
(1)
(2) (3) (4) (5) (6) (7) (8) (9) (10)
(11) (12) (13)

©
2 SIP
EncodeQ 93398 1680 0 217 854448300 0 466 4
773098086 0 2256983835
4 SIP
DecodeQ 103692 275 0 183 773075987 0 6 4
773458450 0 2257346653
5 SipRedirectSessionManager (inputAdapter/rtQueryManager) thread
#0 29036 1271 0 1537 1303331261 0 1 3
773781135 0
Although all queue statistics serve a purpose, the following have a direct impact on call
processing and should be monitored.
SipRedirectSessionManager – These queue statistics refer to the Network Server call
processing thread. This thread processes the incoming INVITEs and provides a
redirection response. A delay in this queue would translate to a delay in
inbound/outbound PSTN calls.
SIP EncodeQ – This queue statistic is related to the encoding and building of outgoing
SIP messages. A delay in this queue results in outgoing messages being backed up.
SIP DecodeQ – This queue statistic is related to the decoding of incoming SIP messages.
A delay in this queue results in incoming messages being backed up.
The two parameters to monitor for each queue are as follows:
bwSystemInternalQueueTimeAvg – The average queue holding time. Note that this
value is in 1/1000 milliseconds, that is, divide by 1000 for the value in milliseconds. It is
recommended that it should be polled and reset once every 15 minutes. High average
queue delays during server busy hour could be an indication of a performance problem.
bwSystemInternalQueueTimeMax – The queue delay high-water mark. It should be
polled and reset once every 15 minutes. Consistent hourly high-water marks (for
example, four polling intervals) of more than one second or any single high-water mark of
more than five seconds should be reported to BroadSoft Support.
bwSystemInternalQueueTimeAvg F NS P Yes. For acceptable queue time

ranges, see BroadWorks System
Engineering Guide.
bwSystemInternalQueueTimeMax F NS P Yes. For acceptable queue time

ranges, see BroadWorks System
Engineering Guide.
20.2.2.2 System Call Processing Statistics

The Network Server acts as a SIP redirect server and as such is stateless. The Network
Server receives INVITEs, processes them, and returns a redirection response (“302
Moved Temporarily” in a successful transaction). Network Server call processing capacity
can be calculated using the BroadWorks System Capacity Planner, which is an Excel
spreadsheet. The basic variable used to describe Network Server capacity is:
calls per seconds (CPS)/transactions per second (TPS).
There are call processing performance measurements that allow a carrier to monitor
Network Server performance. They are also used to perform basic validation on the call
model used in the BroadWorks System Capacity Planner.

©
20.2.2.2.1 Transaction Processing
System-wide transaction processing statistics are available to help monitor system
performance.
broadworks/nsExecutionServer/processing/
NS_CLI/Monitoring/PM/NetworkServer> get bwNSCallpCallsPerSecond

bwNSCallpCallsPerSecond 3
broadworks/nsExecutionServer/protocol/sip/
NS_CLI/Monitoring/PM/NetworkServer> get bwNSSipStatsInviteIns

*bwNSSipStatsInviteIns 14396
bwNSCallpCallsPerSecond (Key Performance Indicator) – This gauge measures the

number of transactions per second that the Network Server is currently handling. It uses
the time stamps of (up to) the last 100 INVITEs received.
Network Servers are licensed based on the number of transactions per second within a
certain time window. When the capacity is exceeded, the Network Server returns a “503
Service Unavailable”. Although traps are generated when licensing capacity is reached,
this gauge can also be used to trend TPS growth.
SIP Messages Per Second (Key Performance Indicator) – The number of SIP
messages per second (MPS) can be used to gauge if a server’s call patterns have
changed. SIP MPS can be calculated using the following counters:
SIP_MPS = Delta [bwNSSipStatsTcpIns + bwNSSipStatsTcpOuts +
bwNSSipStatsUdpIns + bwNSSipStatsUdpOuts] / Polling_Interval_In _Seconds
bwNSSipStatsInviteIns – This counter measures the number of INVITEs processed by
the Network Server.
This counter should be polled hourly and can be used to trend transactions per hour,
which would be the equivalent of busy hour call attempts (BHCA) on an Application
Server.
bwNSCallpCallsPerSecond F NS T No, trending data only.
SIP_MPS H NS T No, trending data only.
bwNSSipStatsInviteIns H NS T No, trending data only.
20.2.2.2.2 Processing Treatments

The Network Server keeps track of calls that result in a treatment. These internal
treatment messages map back to SIP responses (either a 4XX or 5XX). The list of
system-defined treatments and corresponding SIP error messages can be found under
CLI level NS_CLI/System/CallP/Treatment>.
Example:

©
NS_CLI/Monitoring/PM/NetworkServer> get errorStatTable
errorStatTable:
(1) errStatID
(2) errStatName
(3) errStatNbOccurences
(1) (2) (3)

1 invld 73
2 ndmch 16
3 untmt 0
4 undge 0
5 usrnf 260
6 nrixc 0
7 norte 0
8 dftmt 0
9 carjc 0
10 innpa 9
11 incac 0
12 invdd 0
13 inddd 0
14 inidd 0
15 inldd 0
16 tmpun 0
17 bdreq 0
18 frbdn 0
19 reqto 0
20 nserr 0
21 noimp 0
22 bdgwy 0
23 srvun 0
24 srvto 0
25 verns 0
26 invod 0
27 unadn 8
28 unkgw 15
29 sbovf 0
30 carbk 0
This table can (optionally) be polled once a day for trending purposes.
Not all treatment codes report a problem. For example, invld is pegged when a user dials
an invalid number and therefore it is not representative of a problem, while unkgw would
indicate a problem, since it means that the requesting device is not known to the Network
Server. As such, only occurrences of following treatments must be investigated:
 bdgwy – bad gateway
 bdreq – not acceptable
 frbdn – Network Server refuses to process the request
 noimp – functionality not supported
 nserr – Network Server error
 reqto – Network Server request timeout
 sbovf – service bus re-enter overflow
 srvto – Network Server timeout
 srvun – Network Server unavailable

©
 unkgw – unknown gateway
 verns – SIP version not supported
errStatNbOccurences D NS T No, trending data only.

This information is useful because
some internal treatments are an
indication of a problem (for example,
unkgw).
20.2.2.2.3 Request per Device

The Network Server keeps track of requests received from each device.
Example:
NS_CLI/Monitoring/PM/NetworkServer> get neStatTable

neStatTable:
(1) neStatID
(2) neStatName
(3) neStatNbSIPRequests
(4) neStatNbSIPRequestsFailures
(5) neStatNbMSSRequests
(6) neStatNbMSSRequestsFailures
(1) (2) (3) (4) (5) (6)

1 10.7.7.7 0 0 0 0
3 13.39.208.196 0 0 0 0
4 13.39.208.202 4513 7 0 0
5 219.18.125.215 1268 0 0 0
6 2.234.96.51 0 0 0 0
7 2.39.208.203 270 0 0 0
This table can (optionally) be polled once a day for trending purposes.
neStatNbSIPRequests – This counter allows monitoring the amount of traffic the Network
Server is receiving for all known nodes. If useDomainForSubscriberAddress is set to
“true” on the Application Server, the Network Server tracks INVITEs per user domain, not
per Application Server identity.
neStatNbSIPRequests D NS T No, trending data only.

This information is useful to identify
where Network Server requests are
coming from.

©
20.2.3 Application Server, Execution Server, and Profile Server
The Application Server provides a number of SNMP counters and gauges that can be
used to monitor performance. These MIBs can be polled from an external management
system or via the CLI.
When using the IMS mode, Provisioning and Execution activities are split between the
Profile Server and the Execution Server respectively. Counters and gauges are also valid
on the specified server, when indicated.
20.2.3.1 Internal Queue Statistics – Key Performance Indicator
NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).
In general, SIP signaling delays provide a view on any delays that are affecting call
processing. The Application Server does provide Execution Server process internal
queue statistics as well. Even though these delays should be covered in the SIP signaling
delays, monitoring Execution Server internal queue statistics is recommended.
AS_CLI/Monitoring/PM/ApplicationServer> get
-------------------------------------------------------------------------
broadworks/executionServer/systemModule/internalStats/
-------------------------------------------------------------------------
*bySystemInternalQueueResets 0
bwSystemInternalQueueTable:
(1) bwSystemInternalQueueIndex
(2) bwSystemInternalQueueName
(3) bwSystemInternalQueueSize
(4) bwSystemInternalQueueTimeAvg
(5) bwSystemInternalQueueTimeMin
(6) bwSystemInternalQueueTimeMax
(7) bwSystemInternalQueueTimeMaxTimestampMSB
(8) bwSystemInternalQueueLengthCurrent
(9) bwSystemInternalQueueLengthAvg
(10) bwSystemInternalQueueLengthMax
(11) bwSystemInternalQueueLengthMaxTimestampMSB
(12) bwSystemInternalQueueLengthMaxTimestampLSB
(13) bwSystemInternalQueueTimeMaxTimestampLSB
(1) (2) (3) (4) (5)

(6)
(7) (8) (9) (10) (11) (12) (13)
1 CallP ThreadedDBAccessQueue 21 24285 9
103
268 0 0 0 268 2255816335 2257044192
2 Call Half Input Adapter 0 108 95731 0
866
268 0 1685 11 268 2253911749 2253912586
3 Call Half Input Adapter 1 210 47838 0
916
268 0 658 11 268 2253911780 2257044446
4 Registration Input Adapter 799 3702 0
308
268 0 14 3 268 2258153986 2253914110

©
Although all queue statistics serve a purpose, the following primary queues have a direct
impact on call processing and they should be monitored:
Call Half Adapters – These queue statistics refer to the Execution Server call half thread
queues. Each Application Server has at least two call half adapter threads. A delay in
one thread corresponds to a delay in the call that that adapter was handling. If adapters
are delayed, then messaging queues until the delay ends.
SIP EncodeQ – This queue statistic is related to the encoding and building of outgoing
SIP messages. A delay in this queue results in outgoing messages backing up.
SIP DecodeQ – This queue statistic is related to the decoding of incoming SIP messages.
A delay in this queue results in incoming messages backing up.
MGCP EncodeQ – This queue statistic is related to the encoding and building of outgoing
MGCP messages. A delay in this queue results in outgoing messages backing up.
MGCP DecodeQ – This queue statistic is related to the decoding of incoming MGCP
messages. A delay in this queue results in incoming messages backing up.
The two parameters to monitor for each queue are as follows:
bwSystemInternalQueueTimeAvg – The average queue holding time. Note that this
value is in 1/1000 of a millisecond, that is, divide by 1000 to get the value in milliseconds.
This should be polled and reset once every 15 minutes. An average queue delay during a
server busy hour of more than 200 milliseconds would be an indication that this server has
reached server capacity.
bwSystemInternalQueueTimeMax – This is the queue delay high-water mark. For the
call half adapter queues, a high-water mark delay of 2.5 seconds or greater results in the
generation of a bwThreadDelayDetected trap.
This should be polled and reset once every 15 minutes.
20.2.3.2 System Call Processing Statistics

Application Server call processing capacity can be calculated using the BroadWorks
System Capacity Planner, which is an Excel spreadsheet. The basic variables used to
describe Application Server capacity are as follows:
 Number of users
 Busy hour call attempts
 Calls per second
 Simultaneous active calls
There are a number of call processing performance measurements available, which allow
a carrier to monitor Application Server performance and they perform a basic validation on
the call model used in the BroadWorks System Capacity Planner.

©
20.2.3.2.1 Number of Users
The number of provisioned users can be polled.
Example
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/systemModule/systemStats/
AS_CLI/Monitoring/PM/ApplicationServer> get bwNumberOfUsers

bwNumberOfUsers 24739
bwNumberOfUsers – This gauge provides an indication of the number of provisioned

users on the Application Server.
bwNumberOfUsers D AS T No, trending data only.
20.2.3.2.2 Call Originations/Terminations
System-wide call processing statistics are available to help monitor system performance.
Example:
broadworks/executionServer/callpModule/callpStats/
bwCallpNetworkOriginationAttempts
bwCallpUserOriginationAttempts
*bwCallpUserOriginationAttempts 3008
bwCallpUserTerminationsAnswered
*bwCallpUserTerminationsAnswered 2542
bwCallpNetworkOriginationAttempts
bwCallpNetworkTerminationAttempts
*bwCallpNetworkTerminationAttempts 3454
bwCallpNetworkTerminationsAnswered
*bwCallpNetworkTerminationsAnswered 1971
bwCallpUserOriginationAttempts – This counter represents the number of BroadWorks

user origination attempts.

©
bwCallpNetworkOriginationAttempts – This counter represents the number of incoming
calls to BroadWorks users from network devices (for example, PSTN gateway or
softswitch).
bwCallpUserTerminationAttempts – This counter represents the number of calls that
terminated to a BroadWorks user.
bwCallpNetworkTerminationAttempts – This counter represents the number of calls
that terminated to a network device (for example, PSTN gateway or softswitch).
bwCallpUserTerminationsAnswered – This counter represents the number of calls to
BroadWorks users that are answered.
bwCallpNetworkTerminationsAnswered – This counter represents the number of calls
to network devices that are answered (for example, PSTN calls).
Although all of the above counters provide information, the basic monitoring statistic that is
derived is the busy hour call attempts (BHCA).
NbCallAttempts – This is the sum or all access-side and network-side originations that
can be used to calculate the overall number of call attempts the system is handling,
where:
NbCallAttempts = bwCallpUserOriginationAttempts + bwCallpNetworkOriginationAttempt
Polling these counters hourly would provide the BHCA values for the entire day and allow
a carrier to validate the BHCA assumption used in the BroadWorks System Capacity
Planner.
bwCallpNetworkOriginationAttempts F AS T No, trending data only.
bwCallpUserOriginationAttempts F AS T No, trending data only.
bwCallpNetworkTerminationAnswered H AS T No, trending data only.
bwCallpUserTerminationAnswered H AS T No, trending data only.
bwCallpUserTerminationAttempts H AS T No, trending data only.
bwCallpNetworkTerminationAttempts H AS T No, trending data only.
NbCallAttempts = F AS T No, trending data only.

[bwCallpUserOriginationAttempts + Multiply by four to calculate the
bwCallpNetworkOriginationAttempt] BHCA for the server.
20.2.3.2.3 Current/Active Call Statistics
Two current or active call gauges can be monitored.

Example:
broadworks/executionServer/callpModule/callpStats/
AS_CLI/Monitoring/PM/ApplicationServer> get bwCallpCallsPerSecond

bwCallpCallsPerSecond 5

©
AS_CLI/Monitoring/PM/ApplicationServer> get bwCallpActiveCalls
bwCallpActiveCalls 234
bwCallpCallsPerSecond – This gauge measures the rate at which network origination

attempts and user origination attempts occur. It uses the time stamps of up to the last 100
network origination attempts and user origination attempts to calculate the rate.
Polling this gauge every quarter hour can also identify any call processing spikes. A
threshold alarm can be created to send a trap when CPS hits a certain threshold.
Overload controls can be enabled to help mitigate any attack.
bwCallpActiveCalls – This gauge indicates the total number of active sessions. An
active session is defined as an active originating session.
A list of all users involved in active calls can be generated by service provider or enterprise
name, using the Application Server CLI level as follows.
AS_CLI/System/Util/Diag> list svcProviderId <SP/ENT_name>
bwCallpCallsPerSecond F AS T No, trending data only.
bwCallpActiveCalls F AS T No, trending data only.
20.2.3.3 OCI Provisioning Usage
NOTE: This section is valid for the Profile Server as well (Provisioning application).
Release 13.0 introduced the open client interface (OCI) for unified provisioning. All
provisioning transactions initiated regardless of source (web, CLI, or external third party
OCI client) go through the OCI. There are performance measurements that can be
monitored to track provisioning performance.
Example:
broadworks/provisioningServer/psOciModule/psOciStats/
AS_CLI/Monitoring/PM/ApplicationServer> get psOciStatsRequestResponseTime

psOciStatsRequestResponseTime 114
psOciStatsMaxRequestResponseTime
*psOciStatsMaxRequestResponseTime 6685
OCI Messages Per Second (Key Performance Indicator) – OCI MPS rate can be used
to identify changes in provisioning traffic.
OCI_MPS= Delta [psOciStatsNbUpdateRequests + psOciStatsNbQueryRequests +
psOciStatsNbAuthorizationRequests] / Polling_Interval_In_Seconds
psOciStatsRequestResponseTime – This gauge indicates the average time (in
milliseconds based on a rolling average of the last 100 samples) it takes to process an
OCI request.

©
Polling this gauge hourly provides an indication of OCI performance.
The expected average response time depends on the platform; generally, it should be less
than 300 milliseconds under normal operating conditions. Continuous average response
times of more than 500 milliseconds should be reported to BroadSoft Support.
psOciStatsMaxRequestResponseTime – This gauge provides the high-water mark for
OCI response delay.
This gauge can be polled hourly and reset, to provide a view of the maximum delay the
OCI interface has experienced within that polling window.
It is normal that at times the maximum delay (certainly on a small system configuration
such as a V120) is in the range of seconds. Repeated maximum delays polled in the tens
of seconds should be reported to BroadSoft Support.
You can poll psOciStatsMaxRequestName to get the actual request type that caused the
high-water mark delay.

Measurement
OCI_MPS H AS T No, trending data only.
psOciStatsRequestRespons H AS P Yes. If the result is more than 300

eTime milliseconds (low alarm) or if more
than 500 milliseconds (high alarm).
psOciStatsMaxRequestResp H AS P Yes. When the result is more than

onseTime five seconds for four polling intervals
(high alarm).
20.2.3.4 Messaging Statistics
There are SIP-related and MGCP performance measurements that can be used to
monitor system performance.
20.2.3.4.1 SIP/MGCP Messaging Rates

Normal call originations and terminations monitored by BHCA are not the only type of
messaging that impact system capacity.
SIP Messages Per Second (Key Performance Indicator) – SIP MPS rate can be used
to identify changes in SIP traffic patterns. SIP MPS can be calculated using the following:
SIP_MPS= Delta [bwSipStatsTcpIns + bwSipStatsTcpOuts + bwSipStatsUdpIns +
bwSipStatsUdpOuts] / Polling_Interval_In _Seconds
MGCP Messages Per Second (Key Performance Indicator) – MGCP MPS rate can be
used to identify changes in MGCP traffic patterns. SIP MPS can be calculated using the
following:
MGCP_MPS= Delta [bwMGCPStatsMGCPCommandOuts +
bwMGCPStatsMGCPCommandIns + MGCPStatsMGCPResponseOuts +
bwMGCPStatsMGCPResponseOuts] / Polling_Interval_In _Seconds

©
Measurement
SIP_MPS H AS T No, trending data only.
MGCP_MPS H AS T No, trending data only.
20.2.3.4.2 Ancillary SIP Messaging

Normal call originations and terminations monitored by BHCA are not the only type of
messaging that impact system capacity. A system with a high number of ancillary SIP
messages impacts performance (for example, high CPU or call processing delays) without
any calls in progress. SIP REGISTER, SUBSCRIBE/NOTIFY, and OPTIONS message
rates must be monitored so that the system is not flooded.
broadworks/executionServer/sipModule/sipStats/
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsRegisterIns

*bwSipStatsRegisterIns 109926
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsOptionsIns

*bwSipStatsOptionsIns 12424
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsNotifyIns

*bwSipStatsNotifyIns 851
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsNotifyOuts

*bwSipStatsNotifyOuts 574471
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSubscribeIns

*bwSipStatsSubscribeIns 210491
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSubscribeOuts

*bwSipStatsSubscribeOuts 77
20.2.3.4.2.1 SIP Registration Traffic

Excessive SIP registration traffic can have a serious impact on server performance. The
BroadWorks System Capacity Planner considers this by having the SIP registration
interval and SIP authentication as input variables.
Generally, a SIP device’s registration interval towards the Application Server should be 60
minutes or greater, which would mean no more than four REGISTER messages per user
per hour. The four REGISTER messages per user, per hour is derived from the standard
model of a user with SIP authentication enabled and a device registering before the 60
minute period expires. If registrations must be more frequent for NAT traversal, a session
border controller should be used to front the high registration traffic.
The following performance measurement can be used to monitor registration traffic:
bwSipStatsRegisterIns – This counter reflects the total number of REGISTER requests
received by BroadWorks and it can be used to monitor registration traffic.
Monitoring this counter every 15 minutes can help identify “bursty” registration traffic. The
results should be relatively the same throughout the day and should only increase if the
number of SIP registering devices increase.
Generally, each quarter hour reading of this counter should be within 20% of the average
15-minute result.

©
Excessive “bursty” registration traffic patterns should be investigated to identify the root
cause.
The hourly result of bwSipStatsRegisterIns can be used to see if the registration traffic is in
line with the expected registration traffic. A high value of registrations per user has a
scaling impact on a server.
Ideally, registrations per SIP registering user in any given hour
(bwSipStatsRegisterIn/bwNumberOfUsers) should equal to four or less. In general, a
value of more than 10 should be investigated.
The hourly result of bwSipStatsRegisterIns can be used to estimate the impact of
registration traffic in a BHCA equivalency fashion. In general, from a messaging
perspective, a registration and response is equivalent to 2/16 of a call.
For example, an hourly bwSipStatsRegisterIns of 200 KB would be equivalent to (200 KB
* 2/16) ~25 KB BHCA.
20.2.3.4.2.2 SIP OPTIONS Traffic
The SIP OPTIONS method is used by some session border controllers to “ping”
BroadWorks Application Servers. The following performance measurement can be used
to monitor OPTIONS traffic”
bwSipStatsOptionsIns/ bwSipStatsOptionsOuts – This counter reflects the total
number of OPTIONS requests received by BroadWorks and can be used to monitor
OPTIONS traffic.
Ideally, SIP OPTIONS traffic from session border controllers should be no lower than one
message every 45 seconds. This allows the OPTIONS session to terminate. OPTIONS
pinging at a rate below 40 seconds may result in the session remaining active and a
bwAbnormalCallTermination trap when the session audit runs.
The hourly result of bwSipStatsOptionsIns + bwSipStatsOptionsOuts can be used to
estimate the impact of OPTIONS traffic in a BHCA equivalency fashion. In general, from a
messaging perspective, a registration and response is equivalent to 2/16 of a call.
For example, an hourly OPTIONS traffic of 100 KB would be equivalent to (100 KB * 2/16)
~12 KB BHCA.
20.2.3.4.2.3 SIP SUBSCRIBE/NOTIFY Traffic
SIP SUBSCRIBE/NOTIFY messages are used between the Application Server and many
SIP devices to support advanced call control (auto-answer, hold event reporting, off-hook
event reporting). These messages can affect overall performance and they should be
monitored.
bwSipStatsSubscribeIns/bwSipStatsSubscribeOuts – These counters reflect the total
number of SUBSCRIBE traffic in/out of the Application Server.
bwSipStatsNotifyIns/bwSipStatsNotifyOuts – These counters reflect the total number
of NOTIFY traffic in/out of the Application Server.
Monitoring the counters every hour can help identify SUBSCRIBE/NOTIFY traffic impact
on overall performance.
The hourly result of bwSipSubscribeIns + bwSipSubscribeOuts can be used to estimate
the impact of SUBSCRIBE traffic in a BHCA equivalency fashion. In general, from a
messaging perspective, a SUBSCRIBE and response is equivalent to 2/16 of a call.
For example, an hourly bwSipSubscribeIns + bwSipSubscribeOuts of 100 KB would be
equivalent to (100 KB * 2/16) ~12 KB BHCA.

©
The hourly result of bwSipNotifyIns + bwSipNotifyOuts can be used to estimate the impact
of NOTIFY traffic in a BHCA equivalency fashion. In general, from a messaging
perspective, a SUBSCRIBE and response is equivalent to 2/16 of a call.
For example, an hourly bwSipSubscribeIns + bwSipSubscribeOuts of 100 KB would be
equivalent to (100 KB * 2/16) ~12 KB BHCA.

Measurement
bwSipStatsRegisterIns H AS T Ideally, registrations per SIP registering

user in any given hour
(bwSipStatsRegisterIn/bwNumberOfUsers)
should be equal to four or less.
The hourly result of any of these ancillary
message PMs can be used to estimate the
impact of registration traffic in a BHCA
equivalency fashion. In general, from a
messaging perspective, a registration and
response is equivalent to 2/16 of a call.
[bwSipStatsOptionsIns + H AS T No, trending data only.

bwSipStatsOptionsOuts]
[bwSipStatsSubscribeIns+b H AS T No, trending data only.

wSipStatsSubscribeOuts]
[bwSipStatsNotifyIns+bwSi H AS T No, trending data only.

pStatsNotifyOuts]
20.2.3.4.3 SIP Signaling Delays

There are call processing delay-related performance measurements that can be used to
monitor overall performance.
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSetupSignalDelay

bwSipStatsSetupSignalDelay 13
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsMaxSetupSignalDelay

*bwSipStatsMaxSetupSignalDelay 594
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsAnswerSignalDelay

bwSipStatsAnswerSignalDelay 17
AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsMaxAnswerSignalDelay

*bwSipStatsMaxAnswerSignalDelay 255
bwSipStatsSetupSignalDelay – This gauge is applicable to SIP-originated calls. It

measures the time (in milliseconds, based on a rolling average of up to 100 samples) it
takes between the receipt of an INVITE message for the origination of a new call and the
transmission of an INVITE (SIP terminator) to the primary device of the original called
party (in the case of intra-group call) or to the network element of the original called party
(in the case of a call to the PSTN). Delays incurred by a query to an external element (for
example -dip to the Network Server, CNAME query) are part of the measurement. This
measurement is intended to reflect the elapsed delay between receipt of the call setup
signal from the caller and transmission of the call setup signal to the called party.

©
This gauge should be polled every 15 minutes to get a view of the average SIP setup
delays. The average delay should be no more than 200 milliseconds during busy hour.
Averages higher than 200 milliseconds during busy hour are an indication that the server
is performing above capacity.
bwSipStatsMaxSetupSignalDelay – This gauge measures the longest SIP setup-signal
delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of high-water SIP
setup-signal delay. It is not abnormal to see a spike in SIP setup-signal delay. An
occasional reading of one or two seconds is not indicative of a problem. Constant high-
water marks of more than two seconds should be reported to BroadSoft Support.
bwSipStatsAnswerSignalDelay – This gauge measures the time (in milliseconds, based
on a rolling average of up to 100 samples) between the receipt of a 200 OK message
indicating answer and the transmission of a 200 OK indicating answer to the originator.
This measurement is intended to reflect internal queuing, scheduling, and processing
delays. This gauge is applicable only to the SIP-originated calls for which the setup-signal
delay is measured. Answer signal delay is not measured when the call has been
forwarded, redirected, or when the call has encountered route advancing.
This gauge should be polled every 15 minutes to get a view of the average SIP answer
bwSipStatsMaxAnswerSignalDelay – This measures the longest SIP answer-signal
delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of high-water SIP
answer-signal delay. It is not abnormal to see a spike in SIP answer-signal delay. An
bwSipStatsSetupSignalDelay F AS P Yes. If the result is more than

200 milliseconds (medium alarm)
or if more than 500 milliseconds
(high alarm).
bwSipStatsMaxSetupSignalDelay F AS P Yes. If the result is more than

2.5 seconds for four polling
intervals (high alarm).
bwSipStatsAnswerSignalDelay F AS P Yes. If the result is more than

200 milliseconds (medium alarm)
or if more than 500 milliseconds
(high alarm).
bwSipStatsMaxAnswerSignalDelay F AS P Yes. If the result is more than

2.5 seconds for four polling

©
20.2.3.4.4 MGCP Signaling Delays
There are call processing delay-related performance measurements that can be used to
monitor MGCP overall performance.
broadworks/executionServer/mgcpModule/mgcpStats/
AS_CLI/Monitoring/PM/ApplicationServer> get bwMGCPStatsDialToneDelay
*bwMGCPStatsDialToneDelay 2
bwMGCPStatsMaxDialToneDelay
*bwMGCPStatsMaxDialToneDelay 578
bwMGCPStatsSetupSignalDelay
*bwMGCPStatsSetupSignalDelay 89
bwMGCPStatsMaxSetupSignalDelay
*bwMGCPStatsMaxSetupSignalDelay 4276
bwMGCPStatsAnswerSignalDelay
bwMGCPStatsAnswerSignalDelay 17
bwMGCPStatsMaxAnswerSignalDelay
*bwMGCPStatsMaxAnswerSignalDelay 255
bwMGCPStatsDialToneDelay – This gauge measures the time (in milliseconds, based

on a rolling average of up to 100 samples) between receipt of NTFY indicating off-hook
and transmission of RQNT indicating dial tone and request for digits.
This gauge should be polled and reset every 15 minutes to get a view of the average
MGCP setup delays. The average delay should be no more than 50 milliseconds during
busy hour. Averages higher than 50 milliseconds during busy hour are an indication that
the server is performing above capacity.
bwMGCPStatsMaxDialToneDelay – This measures the longest MGCP dial-tone delay
sampled since the system was started or since this measurement was cleared.
This gauge should be polled and reset every 15 minutes to get a view of the high-water
SIP setup signal delay. It is not abnormal to see a spike in a MGCP setup-signal delay.
An occasional reading of one or two seconds is not indicative of a problem. Constant
high-water marks of more than two seconds should be reported to BroadSoft Support.
bwMGCPStatsSetupSignalDelay – This gauge is applicable to MGCP-originated calls.
It measures the time (in milliseconds, based on a rolling average of up to100 samples) it
takes between the receipt of a NTFY message with digits dialed for the origination of a
new call and the transmission of an INVITE (SIP terminator), RQNT (MGCP terminator
with in-band ringback), or CRCX (MGCP terminator without in-band ringback) to the
primary device of the original called party (for an intra-group call) or to the network element
of the original called party (for a call to the PSTN). Delays incurred by a dip to the Network
Server are removed from the measurement. This measurement is intended to reflect
internal queuing, scheduling, and processing delays.

©
This gauge should be polled and reset every 15 minutes to get a view of the average
MGCP setup delays. The average delay should be no more than 200 milliseconds during
busy hour. Averages higher than 200 milliseconds during busy hour are an indication that
the server is performing above capacity.
bwMGCPStatsMaxSetupSignalDelay – This gauge measures the longest MGCP setup-
signal delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of a high-water
MGCP setup-signal delay. It is not abnormal to see a spike in a MGCP setup-signal
delay. An occasional reading of one or two seconds is not indicative of a problem.
Constant high-water marks of more than two seconds should be reported to BroadSoft
Support.
bwMGCPStatsAnswerSignalDelay – This gauge measures the time (in milliseconds,
based on a rolling average of up to 100 samples) between the receipt of a 200 OK
message indicating answer or a NTFY off-hook, indicating answer and the transmission of
a MDCX indicating answer to the originator. This measurement is intended to reflect
internal queuing, scheduling, and processing delays. This gauge is applicable to only the
MGCP-originated calls for which the setup-signal delay is measured. Answer signal delay
is not measured when the call has been forwarded, redirected, or when the call has
encountered route advancing.
This gauge should be polled every 15 minutes to get a view of the average MGCP answer
bwMGCPStatsMaxAnswerSignalDelay – This measures the longest MGCP answer-
signal delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of high-water
MGCP answer delay. It is not abnormal to see spike in MGCP answer delay. An
bwMGCPStatsSetupSignalDel F AS P Yes. If the result is more than 200

ay milliseconds (medium alarm) or if
more than 500 milliseconds (high
alarm).
bwMGCPStatsMaxSetupSignal F AS P Yes. If the result is more than 2.5

Delay seconds for four polling intervals
(high alarm).
bwMGCPStatsAnswerSignalDe F AS P Yes. If the result is more than 200

lay milliseconds (medium alarm) or if
alarm).
bwMGCPStatsMaxAnswerSign F AS P Yes. If the result is more than 2.5

alDelay seconds for four polling intervals
(high alarm).
bwMGCPStatsDialToneDelay F AS P Yes. If the result is more than 50

milliseconds (medium alarm) or if
alarm).

©
20.2.3.4.5 SIP Network Element Retry Percentage
The Application Server keeps track of SIP retry percentages to network-side nodes such
as Media Servers, Network Servers, other Application Servers, and PSTN interconnection
nodes (softswitch or network gateways).
bwSipStatsMsgRetryToNeTable
bwSipStatsMsgRetryToNeTable:
(1) bwSipStatsMsgRetryToNeID
(2) bwSipStatsMsgRetryToNeAddr
(3) bwSipStatsMsgRetryToNePercentage
(1) (2) (3)

1 14.14.208.196 0
2 15.15.125.217 51
3 14.14.208.202 2
4 15.15.125.215 60
5 14.14.208.195 0
6 15.15.125.216 52
7 14.14.208.203 0
8 14.14.208.204 0
9 14.14.208.230 0
10 14.14.208.194 0
11 15.15.125.218 8
12 14.14.208.207 0
13 14.14.208.200 0
14 15.15.29.76 0
This table should be polled daily to get a view of retry percentages to various nodes. A
retry percentage more than two percent to any network-side node should be investigated.
Session border controllers show up in the list as well. The retry percentage to these
nodes should be ignored since they include retries to access devices.
bwSipStatsMsgRetryToNe D AS P Yes. If the result is more than

Percentage 3% (medium alarm) or if the
result more than 5% (high
alarm).
20.2.3.4.6 SIP Message Responses
The Application Server tracks responses to all SIP methods received and sent. This can
be useful in “trending” responses to monitor for changes in call processing behavior (that
is, a sudden increase in a specific negative response).

©
Example:
bwSipStatsInviteResponsesTable
bwSipStatsInviteResponsesTable:
(1) bwSipStatsInviteResponseCodeValue
(2) bwSipStatsInviteResponseIns
(3) bwSipStatsInviteResponseOuts
(1) (2) (3)

100 11554 6254
180 2754 3573
181 0 0
182 0 0
183 4386 2208
200 13185 5886
300 0 0
301 0 0
302 9466 0
305 0 0
380 0 0
400 2 14
401 0 0
402 0 0
403 0 0
404 180 0
405 0 0
406 10 0
407 1 0
408 1 40
409 0 0
410 3 0
411 0 0
413 0 0
414 0 0
415 11 0
420 0 0
480 167 36
481 24 6
482 0 0
483 0 0
484 100 0
485 0 0
486 95 44
487 3534 715
488 6 0
500 49 0
501 0 0
502 0 0
503 26 0
504 34 0
505 0 0
600 18 95
603 35 0
604 13 51
606 4 2

©
Although “trending” all messages may be worthwhile, the two SIP methods tables that
should be monitored are bwSipStatsRegisterResponsesTable and
bwSipStatsInviteResponsesTable.
These tables should be polled daily to trend request/response messaging.
For INVITE in/out, sudden increases in “5XX Server Failures” and “6XX Global Failures”
should be investigated. 5XX Server Failures result in traps as well.
For REGISTER in, sudden increases in 404 User Not Found, 401 Unauthorized, and 482
Loop Detected responses should be investigated. Each 401 Unauthorized also results in
a trap.
bwSipStatsInviteResponseIns/ D AS T No, trending data only.

bwSipStatsInviteResponseOuts Trending responses to INVITEs
IN/OUT and REGISTERs IN can
help identify problems (that is,
many 503s coming in, 404
responses to registers).
20.2.3.5 Provisioning Server BCCT Queue Statistics
NOTE: This section is valid for the Profile Server as well (Provisioning application).
Release 13.0 introduced the BroadWorks Common Control Transport (BCCT) protocol.
BCCT is used to transport OCS to Application Server OSS/OCI messages, Application
Server to Network Server NSSYNC messages, third-party OCI provision systems, and
some internal functions such as protocol monitoring. BCCT queue statistics can optionally
be monitored.
Example:
AS_CLI/Monitoring/PM/ApplicationServer> get -r
-------------------------------------------------------------------------
-----
broadworks/provisioningServer/psSystemModule/psInternalStats/
-------------------------------------------------------------------------
-----
*psSystemInternalQueueResets 0
psSystemInternalQueueTable:
(1) psSystemInternalQueueIndex
(2) psSystemInternalQueueName
(3) psSystemInternalQueueSize
(4) psSystemInternalQueueTimeAvg
(5) psSystemInternalQueueTimeMin
(6) psSystemInternalQueueTimeMax
(7) psSystemInternalQueueTimeMaxTimestampMSB
(8) psSystemInternalQueueLengthCurrent
(9) psSystemInternalQueueLengthAvg
(10) psSystemInternalQueueLengthMax
(11) psSystemInternalQueueLengthMaxTimestampMSB
(12) psSystemInternalQueueLengthMaxTimestampLSB
(13) psSystemInternalQueueTimeMaxTimestampLSB

©
(1) (2) (3) (4) (5) (6) (7)
(8)
(9) (10) (11) (12) (13)
1 OCIReporting Output Adapter 643 2979 0 158 268
0
0 0 268 2254028188 2258305114
psSystemInternalQueueTimeAvg – This BCCTIncomingMessageQueue gauge

provides the average time a message is in the BCCT queue. Note that this value is in
1/1000 of a millisecond, that is, divide by 1000 for the value in milliseconds.
Polling this gauge hourly provides a view on the average time a BCCT message remains
in the BCCT queue prior to processing.
The average BCCT queue time should be less than 100 milliseconds.
psSystemInternalQueueTimeMax – This BCCTIncomingMessageQueue gauge
provides the high-water mark time (in milliseconds) a message is in the BCCT queue.
Polling and resetting this gauge hourly can provide a view of the maximum time a
message was in the BCCT queue.
It is normal that at times the maximum delay (certainly on a small system configuration
such as a V120) is in the range of seconds. Repeated high maximum delays polled
should be reported to BroadSoft Support.
psSystemInternalQueueTimeAvg H AS P Yes. If the result was more than

300 milliseconds (low alarm) or
if more than 500 milliseconds
(high alarm).
psSystemInternalQueueTimeMax H AS P Yes. If the result is more than

five seconds for four polling
20.2.3.6 SMTP Usage
The Application Server uses defined SMTP servers to send e-mail for features such as
Call Notify and Voice Messaging Notification. The Application Server tries the primary
SMTP server; if that server does not respond within 30 seconds, the Application Server
tries the secondary server.
--------------------------------------------------
broadworks/executionServer/smtpModule/smtpStats/
--------------------------------------------------
*bwSMTPTotalPrimaryEmailSendAttempts 559
*bwSMTPTotalSecondaryEmailSendAttempts 0
*bwSMTPPrimaryUnSuccessfulEmailSendAttempts 0
*bwSMTPSecondaryUnSuccessfulEmailSendAttempts 0

©
bwSMTPTotalPrimaryEmailSendAttempts/bwSMTPPrimaryUnSuccessfulEmailSen
dAttempts – These counters are pegged each time the Application Server successfully
sends an e-mail to the SMTP server.
Polling this counter hourly provides a view of Application Server to SMTP server traffic.
Each pegging of bwSMTPPrimaryUnSuccessfulEmailSendAttempts means the primary
SMTP server could not be reached. If this counter is being pegged regularly, the carrier
should investigate why the primary SMTP server did not reply.
bwSMTPPrimaryUnSuccessfulEmailSendAttempts/bwSMTPSecondaryUnSuccessf
ulEmailSendAttempts – These counters are pegged each time the Application Server
times out to the SMTP server or the SMTP server returns an error.
A bwSMTPConnectivityFailure trap is generated each time either of these counters is
pegged.
Every pegging of an unsuccessful e-mail send counter means that the SMTP server could
not be reached or returned an error code. If this counter is pegged regularly, the carrier
should investigate.
Depending on the setup of the SMTP server, unsuccessful e-mail send counters may be
pegged if an invalid e-mail address is set for a user.
bwSMTPTotalPrimaryEmailSendAt H AS T No, trending data only.

tempts
bwSMTPTotalSecondaryEmailSen H AS T No, trending data only.

dAttempts
bwSMTPPrimaryUnSuccessfulEm H AS T No, trending data only.

ailSendAttempts
bwSMTPSecondaryUnSuccessfulE H AS T No, trending data only.

mailSendAttempts
20.2.3.7 Voice Messaging Retrieval Statistics
When using BroadWorks Voice Messaging, the Application Server is responsible for
connecting to the user’s mail server and downloading messages that are played back via
the voice portal. Each time a user logs in to the voice portal, the Application Server
connects as a mail client to the user’s mail account using the provisioned credentials.
Voice messaging retrieval statistics can be monitored.
Example:
-------------------------------------------------------------------------
-----
broadworks/executionServer/imsModule/imsStats/
-------------------------------------------------------------------------
-----
*bwIMSSuccessfulConnectionAttempts 285
*bwIMSUnsuccessfulConnectionAttempts 4
*bwIMSSuccessfulDownLoadAttempts 350

©
*bwIMSUnsuccessfulDownLoadAttempts 0
bwIMSUnsuccessfulConnectionAttempts – This counter is pegged when the

Application Server is unable to connect to a user’s mail server account as a mail client.
A bwIMSConnectionFailure trap is generated for every connection failure.
The unsuccessful connection is generally related to improper voice messaging
configuration for that user (for example, incorrect user ID or password). A protocol monitor
capture of the user’s attempt to retrieve messages should identify the problem.
When attempting to listen to the voice mail via the voice portal, the user hears the
announcement “This operation cannot be completed at this time”.
This counter should be polled periodically (for example, once a day) to track connection
failures. Ideally, the result should be “0”.
bwIMSUnsuccessfulDownLoadAttempts – This counter is pegged when the
Application Server is unable to download a message from a user’s mail server account.
A bwIMSRetrieveMessageError trap is generated for every retrieval failure.
This counter should be polled periodically (for example, once a day) to track connection
failures. Ideally, the result should be “0”.
If there are a high number of these failures daily, the carrier should investigate using the
generated trap time stamp as a reference to trace back into the Application Server
Execution Server logs.
bwIMSSuccessfulConnection H AS T No, trending data only.

Attempts
bwIMSUnsuccessfulConnecti H AS T No, trending data only since an alarm

onAttempts is generated.
Even a peg of one means that
someone has incorrectly configured
voice mail. No alarm should be raised,
but ideally, this number should be
zero.
bwIMSSuccessfulDownLoadA H AS T No, trending data only.

ttempts
bwIMSUnsuccessfulDownLoa H AS T No, trending data only since an alarm

dAttempts is generated.
Pegging in this case means that the
Application Server attempted to
download what was considered a valid
e-mail and it failed.

©
20.2.3.8 Database Usage
Application Server Provisioning Server and Execution Server database update and query
usage statistics can be “trended”.
Broadworks/executionServer/databaseModule/databaseStats/timesTen/
-------------------------------
*bwXSUpdateCount 3568
*bwXSQueryCount 43165
broadworks/provisioningServer/psDatabaseModule/psDatabaseStats/psTimesTen
--------------------------------
*bwPSUpdateCount 6702
*bwPSQueryCount 21992
bwXSUpdateCount/bwPSUpdateCount – These counters represent the number of

database updates initiated by the Provisioning and Execution Server processes.
XSQueryCount/bwPSQueryCount – These counters represent the number of database
queries initiated by the Provisioning and Execution Server processes.
bwXSUpdateCount/bwPSUpdateCount H AS T No, trending data only.
bwXSQueryCount/bwPSQueryCount H AS T No, trending data only.
20.3 BroadWorks SNMP Trap Monitoring

BroadWorks supports SNMPv2 and SNMPv3. Traps generated by all servers, should be
collected, and monitored using the service provider’s network management station (NMS).
For information on how to configure SNMP as well as a list of all BroadWorks traps, see
the BroadWorks Fault and Alarm Interface Specification along with each server’s specific
fault MIB.
There are hundreds of BroadWorks traps. Not all traps are indicative of a problem.
BroadWorks traps range in severity from Informational to Critical. Trap severity is based
on the impact on the system operations as a whole and should be dealt with according to
severity. Trap severities are a generalized view of impact on the system. Low and
medium severity traps should not be disregarded unless the impact is understood.
Although all traps provide value, as a best practice, the following sections provide a view
of the minimum subset of traps that should be monitored.
20.3.1 Common BroadWorks Server Traps

These traps are generated as part of the server scheduled tasks.
bwSystemHealthReport:
 Sent when healthmon task has been activated.
 Healthmon checks all BroadWorks component and disk space.

 Should be scheduled to run every 30 minutes.
 A trap is sent every 30 minutes. Notification severity indicates normal server
operation.

©
bwCPUIdleTimeLimitReached and bwMemoryLimitReached:
 Sent when the CPUMon task has been activated.
 CPUMon monitors server CPU usage, memory, and paging.

 Should be scheduled to run every six minutes.
 A trap is only sent if any of the monitored values surpass the thresholds defined in the
CPUMon script.
 Threshold levels can be modified by changing the bwCPUMon script.
bwDatabaseSyncReport:
 Sent when the synchcheck_basic.pl task has been activated on Application Servers,
Network Servers, and Messaging Servers.
 The synchcheck_basic.pl does a quick non-service impacting comparison of the
number of table rows in two peer databases.
 Should be scheduled to run once a night (off-peak hours) on the secondary server.
Do not activate on the primary server.
 A trap is only sent whenever the script executes. It is notification severity when
databases are the same and critical severity when a discrepancy is detected.
NOTE: Before escalating this trap to BroadSoft Support, manually run synchcheck_basic.pl
again. It is possible that during the snapshot test, not all replicated information was pushed, or
that tables were in flux.
20.3.2 Server Process Problems

The following critical severity traps are generated on all servers in the event of
BroadWorks process death. The process should be restarted automatically by the
process monitor, but any occurrence of these traps must be escalated to BroadSoft
Support to make sure the root cause of the process death is understood.
 bwPMProvisioningServerDeath
 bwPMExecutionServerDeath
 bwPMNSProvisioningServerDeath
 bwPMNSExecutionServerDeath
 bwPMMediaServerDeath
 bwPMremotexlaDeath
20.3.3 Software Error Traps

The following traps are generated in the event of an internal software error. They do not
necessarily affect the system, but they are an indication of a software problem. These
traps must be escalated to BroadSoft Support to make sure the root cause of the software
error is understood.
 bwGeneralSoftwareError
 msSoftwareError

©
20.3.4 Media Server Traps
Although the Media Server generates a host of traps, the following traps affect the service
and they must be investigated.
msSmtpFailure
 Media Server could not relay the recorded voice messaging e-mail through the
requested SMTP server.
 Usually indicative of a problem on the SMTP server.
msHTTPConnectionFailure, msHTTPReceiveFailure, msHTTPFileNotFound, and

msInvalidAudioFile
 Media Server downloads .wav files (greetings, announcements, or voice mail) for
playback from the Application Server.
 Any of these traps indicate a failure with the download or file in question.
 General result is that the end user does not hear the greeting, announcement, or
voice mail (call failure).
msMemAllocFailure
 Media Server IVR process does not have enough memory.
 Every instance of this trap means that the Media Server could not allocate memory for
the call and the call fails.
 IVR memory can be increased under CLI level MS_CLI/Service/IVR, parameter
memorySize.
msNoPortAvailableErrors
 This trap occurs every time a request comes to the Media Server and it cannot
process it due to licensing limits.
 The Application Server should route advance to the next Media Server.
 From Release 12 and higher, the number of ports a Media Server can support is
calculated based on hardware type. The different port numbers are defined in file
/usr/local/broadworks/bw_base/conf/hardware_cap.csv. If the hardware type is not
known, the Media Server calculates the maximum number of ports based on number
of CPUs and CPU speed.
20.3.5 Network Server Traps

The Network Server does not generate many traps under normal operating conditions.
However, the following traps affect the service and they must be investigated.
bwNSSynchUpdateFailureError
 The Network Server could not process a synchronization request from an Application
Server (for example, to add or delete an enterprise, group, or user).
 The reason text should provide a clear explanation as to why the synchronization
failed. A synchronization failure results in a failure on the Application Server for that
specific task.
 One of the main causes for synchronization errors once an Application Server and
Network Server have been synchronized is attempting to assign a directory number
(DN) to an Application Server user that does not meet the valid NDC format for that
country code.

©
On the Network Server, you can check your specific country code rules under
NS_CLI/System/CallP/CountryCodes/NDC>.
bwLicenseThreshold
 This trap is generated when a Network Server reaches 80% of the license file defined
transactions per time period value.
 Once a Network Server hits the maximum transactions in the defined time period
window, it returns a SIP “503 Service Unavailable” to all SIP requests.
 This condition can be the result of a routing loop between the PSTN and BroadWorks.
For example, the PSTN considers that a number is owned by BroadWorks and
BroadWorks considers that the number is owned by the PSTN and the routingNE has
an off-net policy that causes the call to route back to the PSTN.
bwNSCallGotTreatment
 This informational severity trap is not necessarily indicative of a problem, but it could
be used as a flag whenever the Network Server hits a specific treatment.
 On the Network Server under NS_CLI/System/CallP/Treatment>, you can enable
which treatments result in a trap. By default, all are enabled.
 You may want to disable all traps and then enable only ones for treatments for which
you are most interested. For example, “unkgw” means the Network Server received a
SIP request from a device it does not recognize.
20.3.6 Application Server, Execution Server, and Profile Server Traps

Although the Application Server generates a host of traps, as a best practice, the following
traps should be monitored.
When using the IMS mode, Provisioning and Execution processes are divided between
the Profile Server and the Execution Server. Traps are also valid on the specified server,
when indicated.
20.3.6.1 POP3/IMAP Traps
The Application Server must be able to reach defined SMTP server(s) to support features
such as Call Notify and Call Center Statistics Reporting. If none of the defined SMTP
servers can be reached, a bwSMTPConnectivityFailure trap is generated.
The Application Server Voice Messaging service uses a standard POP3/IMAP protocol to
store user’s voice mail. The Application Server must be connected to the user’s mail
server account to retrieve messages and manage the mailbox. Any one of the following
traps would indicate a problem related to Application Server Voice Messaging service for a
user:
 bwIMSConnectionFailure
 bwIMSRetrieveMessageError
 bwIMSRetrieveMailBoxInfoError
 bwIMSUpdateMailBoxError
 bwIMSCloseConnectionError

©
20.3.6.2 Routing Traps
bwMediaServerNotResponding
This trap is generated when a Media Server fails to respond to an Application Server
request in the allotted time. On the initial INVITE to the Media Server, the Application
Server uses the re-transmission parameters defined under the CLI level
AS_CLI/System/CallP/Routing/MediaServerSelection> MediaServerTimerLength.
By default, this timer is set to 1500 milliseconds. In general, a Media Server should
respond back within 1.5 seconds. Occurrences of this trap must be investigated to see if
the timer needs to be increased for a specific network.
bwMssNetworkServerNotResponding and bwMssNoResponse

The bwMssNetworkServerNotResponding trap, which relates to Application Server MSS
requests to the Network Server, is generated if a Network Server does not respond within
the allotted time. The MSS timer is controlled under the CLI level
AS_CLI/System/CallP/Routing/MediaServerSelection> mssRouteTimerLength.
By default, this timer is set to 800 milliseconds. In general, a Network Server should
respond back within 800 seconds. Occurrences of this trap need to be investigated to see
if the timer needs to be increased for a specific network.
From an operational perspective, a bwMssNetworkServerNotResponding trap is not
service affecting in that the Application Server route advances to the next Network Server.
On the other hand, a bwMssNoResponse trap means that no Network Server returned a
respond and the call failed.
20.3.6.3 Application Server/Network Server Synchronization Traps
These traps are generated on the Application Server in the event of a failure to push
user/group additions/deletions to the Network Server.
bwNSSynchronizationConnectivityFailure
The Application Server could not open a TCP socket (or the established socket failed) to
the Network Server. An occasional occurrence of this trap is not an issue. The
Application Server re-opens the socket to the NetworkServer. However, continuous
occurrence of this trap indicates a connection problem and it must be escalated to
BroadSoft Support.
bwNSSynchronizationFailure
This trap is generated on the Application Server when an attempt to synchronize a
group/user addition/deletion failed. The trap contains detailed information about the
failure. There should be a corresponding trap on the Network Server as well.

©
20.3.6.4 SIP/MGCP Traps
These traps are generated on a per-call basis by the Application Server. Most are
classified as low severity, but a carrier should understand why they are occurring.
bwAuditAbnormalCallTermination
The Application Server audits (send a re-INVITE or UPDATE) every active call. If the
endpoint fails to respond, this trap is generated and the CDR release cause indicates an
AbnormalCallTermination reason. Session audit is controlled at the following CLI levels:
 AS_CLI/System/CallP/SessionAudit> active = true
 AS_CLI/Interface/SIP> sessionAuditAllowed = true
A session audit should NOT be disabled, as this can lead to a hanging session. The
default session audit interval is 10 minutes for every call. The interval is controlled with the
following CLI parameter (in seconds):
 AS_CLI/System/CallP/SessionAudit> interval = 600
If you are experiencing a large number of these traps, it can indicate an interoperability
issue with a specific device type.
bwSipServiceUnavailableReceived
This trap is generated in the event of an endpoint returning a 5xx SIP response. An
endpoint returning a 5xx response (for example, 500 or 503) generally means that the
device could not handle the call and depending on when the 5xx is received either results
in a call failure or the Application Server route advancing to the next contact.
This trap must be investigated by the carrier as it is indicative of a potential issue in the
network (for example, a device problem or no circuits were available on the PSTN
interconnect).
bwSipMaxRetriesExceeded
This trap indicates that the Application Server tried to send a SIP message to a device and
the device did not respond within the allotted time allowed in SIP. This can be considered
normal when a device is no longer available (for example, a SIP phone is unplugged).
When there are a large number of these traps, it must be investigated by the carrier.
bwMGCPUnrecognisedDomainName and bwSipUnrecognisedDomainName

These traps are generated when the Application Server needs to send a message to a
contact that is not an IP address and requires address resolution (for example, DNS,
/etc/hosts) and that the address resolution failed.
These traps can be serious, in that the call processing thread handling that specific call
hangs until the resolution is complete. If the DNS is slow in responding with the failed
lookup, the call processing thread can hang for many seconds.
Upon receiving this trap, the carrier should take action to make sure that the unknown
domain is addressed by either proper address resolution or device reconfiguration.

©
bwMGCPReTransmissionTimeout
This trap indicates that the Application Server tried to send an MGCP message to a device
and the device did not respond. Given that, MGCP integrated access devices (IADs) are
generally static devices and always online, this trap needs to be investigated by the
carrier.
bwSipAuthenticationFailure
This trap is generated when a SIP user has authentication enabled and the authentication
process fails. This is indicative of devices trying to register without the proper credentials.
These should be investigated by the carrier to determine the source device.
20.3.7 Application Server Execution Server Process Problems
The following traps are generated on the Application Server whenever a serious problem
occurs with one of the call processing internal threads. The thread should be restarted
automatically, but any occurrence of these traps needs to be escalated to BroadSoft
Support to make sure the root cause of the thread death is understood.
 bwForcedExitDueToHungThread
 bwCallPThreadAutoRestart
Less serious than a thread death, a call processing thread can be delayed. A delay
can occur for various reasons. The following trap is generated whenever one of the
existing call processing threads is delayed for more than 2.5 seconds.
 bwThreadDelayDetected
Any occurrence of this trap needs to be escalated to BroadSoft Support to make sure
the root cause of the thread death is understood.
20.3.8 Overload Control Monitoring

The Application Server and Network Server support call processing overload controls.
They transition from one zone to another (green, yellow, or red) as call processing delays
increase. Once a server hits the red zone, an overload action occurs (either drops a
message or responds with a “302 Temporarily Moved” response, redirecting the user to
the secondary server). The bwOverloadZoneTransition trap is generated on a zone
transition.
Overload control parameters are configurable using the CLI levels as follows:
 AS_CLI/System/CallP/OverloadControls for the Application Server
 NS_CLI/System/ OverloadControls for the Network Server
20.3.9 Device Operational State Monitoring

Device polling (Application Server/Network Server/Media Server/PGW) can be enabled on
the Application Server and Network Server. When activated, the device monitoring
periodically pings the device with a SIP OPTIONS method to see if the device is available.
Servers poll devices every five minutes (default setting) to establish the operational state.
The following traps are generated (depending on the device type) on a device failure to
respond to SIP ping:
 bwNetworkDeviceIsFailed

©
 bwAccessDeviceIsFailed
Once a device starts to respond to device pings again, the same alarm is sent out with a
status of “clear”.
The online/offline status provided by this feature does not take the device out-of-service
from a routing perspective. The BroadWorks server does not stop sending or receiving
messages from failed devices, but it moves them to the end of any route list. For example,
if a network gateway is not responding to pinging, it is returned as the last contact
regardless of the weighting results returned by call processing.
Polling can be enabled on the Network Server at the following CLI levels by setting the poll
parameter to “true”:
 NS_CLI/System/Device/HostingNE
 NS_CLI/System/Device/RoutingNE
 NS_CLI/System/Device/ResourceNE
Polling can be enabled on the Application server at the following CLI levels:
 AS_CLI/System/Device/NetworkServers
 AS_CLI/System/Device/Monitor/AccessDevice
A best practice is to enable routingNE, hostingNE, and resourceNE device polling on the
Network Server and NetworkServers device polling should be enabled on the Application
Server. Access device polling should not be enabled on dynamically registering devices,
such as SIP phones, since the user can take the device offline by unplugging it. Access
device monitoring should be restricted to static devices, such as IADs or trunking
gateways.
20.4 Miscellaneous Monitoring

The section outlines various miscellaneous items that should be monitored on
BroadWorks servers.
20.4.1 BroadWorks System Sanity

The health of the BroadWorks application should be monitored using the healthmon
command available on all BroadWorks servers. Healthmon can be configured as a
scheduled task that runs at the designated interval and provides a health check of a server
by sending an SNMP trap. It is recommended that the system provider configure
healthmon to run at 30-minute intervals.
Healthmon monitors the following items:
 File system size (disk usage)
 BroadWorks server processes
 SNMP subagents
 On the Application Server, Network Server, and Messaging Server:
− Third-party processes (TimesTen, Tomcat, Apache, RSYNC, and tnameserver)
− Database replication (full two-way replication testing)
− Database replication lagging (for more information, see section 11.1 Application
Server, Network Server, and Messaging Server Database Replication and 11.1.2
TimesTen Replication Monitoring)

©
− Database notification lagging (for more information, see section 11.1 Application
Server, Network Server, and Messaging Server Database Replication and 11.1.2
TimesTen Replication Monitoring)
− Peer-to-peer SSH connectivity
The healthmon script Keep Alive trap can be configured using the command line
interface.
AS_CLI/Maintenance/Scheduler> add healthmon minute 30
...Done
AS_CLI/Maintenance/Scheduler> get
Id Name Date Day Hour Minute
===========================================================
2 healthmon - - - 30
2 entries found.
Under usual operating conditions, healthmon sends a SNMP trap with severity equal to
Informational. If healthmon detects a problem, the trap severity reflects the severity of the
problem and provides recommendation text on how to proceed.
Following is a sample output when replication is not running.
--------------------------------
Date and time : Thu Aug 7 09:56:41 EDT 2003
--------------------------------
Replication: a remote server is not receiving replication updates.
Databases may be out-of-synch.
--------------------------------
Recommendations
---------------
Replication must be started on all remote servers (repctl start). If
databases are out-of-synch, they must be re-synchronized first (with the
importdb.pl tool). Please refer to the BroadWorks Maintenance Guide for
detailed procedures.
--------------------------------
Following is a sample output when database replication lagging and notification lagging
are detected.
--------------------------------
--------------------------------
Recommendations
---------------
Replication is severely lagging on one of the servers in the cluster. If
the system is being operated in standalone mode, turn off replication.
Otherwise, import the database from the server exhibiting the lagging to
ensure prompt data synchronization.

©
Database XLA is severely lagging on one of the servers in the cluster.
If the files held by database notification continue to grow and the LSN’s
do not advance, contact BroadSoft support prior to restarting the
affected server.
--------------------------------
Healthmon, when run with the -d option, provides details on database replication lagging
and database notification lagging. For more information, see the healthmon description in
section 11.1.2 TimesTen Replication Monitoring.
The healthmon script can also be run manually from the UNIX prompt or from the
command line interface under AS_CLI/Maintenance/Tools. Healthmon should be
manually run on all servers after any maintenance task has been performed.
20.4.2 Redundancy Monitoring

The BroadWorks Application Server, Network Server, and Messaging Server redundancy
solution relies on TimesTen database replication to make sure the cluster members’
databases are synchronized. It is critical that replication is not stopped during normal
operations. Stopping TimesTen replication can result in a database out-of-sync condition.
Replication status is automatically monitored by healthmon and problems are reported as
part of the healthmon SNMP trap generation. When healthmon reports that a server has
stopped its database replication for a given peer, this peer’s database must be imported to
re-synchronize it with the cluster.
Replication status can be manually checked from the UNIX prompt using the repctl
status command.
Example:
bwadmin@MTLAS04$ repctl status
-----------------------------
Datastore name = AppServer
Replication State


A Replication Agent policy set to “always” indicates that the replication process is running.
A Replication Agent policy set to “never” indicates that the replication process is not
running. The File Replication pid represents the process ID used by the RSYNC process.

©
Database replication can be manually tested on peer servers by adding data to one server
and ensuring it is present on the other, and then deleting it from the peer, ensuring the
delete operation is replicated back to the original server. For example, on the primary
server from CLI level System/Alias, add an entry called test, verify that it is replicated to
the secondary server, and then on the secondary server, delete the system alias test and
make sure it is deleted from the primary.
File replication can be manually tested on the Application Server by creating a test file
called test.wav on the primary server in /var/broadworks/userfiles/VM and ensuring that it
is replicated to the secondary server within 10 minutes. You can then add a test file on the
secondary server in /var/broadworks/userfiles/VM and verify that it replicates back to the
primary within 10 minutes. Note that you need to use a peercmd rm to delete the files
from both servers at the same time.
For more information on replication state, database replication lagging state, and database
notification lagging state data, see sections 11.1 Application Server, Network Server, and
Messaging Server Database Replication and 11.1.2 TimesTen Replication Monitoring.
20.4.3 Application Server Migrated User Monitoring

Under usual operating conditions, all users should be hosted on the primary Application
Server. However, it is possible that a user temporarily rolls over (migrates) to the
secondary server (for example, for an intermittent IP problem). The user rolls back to the
primary on the next origination or rolls back when the secondary Application Server
Redundancy (ASR) timer expires. This is normal and does not create a problem for a
redundancy system.
However, it is not usual to have a large number of users migrated to the secondary server
while the primary server is up and running, or to have users go between the primary and
secondary servers. This can be an indication of an improper DNS configuration or an IP
routing issue affecting a number of users. It is important to identify and correct the cause
of these unnecessary migrations.
The Application Server provides the nbOfMigratedUsers SNMP gauge to track the
number of migrated users. This is polled by the system provider’s Network Management
System (NMS).
-----------------------------------------------------------------
broadworks/executionServer/redundancyModule/redundancyStats/
------------------------------------------------------------------
nbOfMigratedUsers 2
The system provider should set up threshold alarms to provide SNMP traps when the
number of migrated users crosses a certain threshold. Two threshold alarms should be
configured, one for low severity and the other for medium severity. The low severity trap
provides an indicator for an acceptable amount of migrations. The medium severity trap
provides an indicator that a large number of users have been migrated and should be
investigated.
AS_CLI/Monitoring/Threshold> add gauge "5 users migrated. Verify that

users rollback to primary AS" nbOfMigratedUsers 0 5 low active
AS_CLI/Monitoring/Threshold> add gauge "50 users migrated. This is an

abnormal condition and should be investigated" nbOfMigratedUsers 6 50
medium active
...Done
AS_CLI/Monitoring/Threshold> get gauge

©
Index Description
Name Low High Severity Status
===================================================================
1 5 users migrated. Verify that users rollback to primary AS
nbOfMigratedUsers 0 5 low active
2 50 users migrated. This is an abnormal condition and
should be investigated nbOfMigratedUsers 6 50 medium
active
Depending on network reliability, the high and low notification values may need to be
adjusted based on the service provider’s network.
Once a migration problem has been identified, the actual migrated users can be identified
using the CLI under the System/Redundancy/MigratedUsers level. This information can
be helpful to identify the root cause of the problem.
AS_CLI/System/Redundancy/MigratedUsers> get
User ID
======================
fred@vox.com
mkushnir@broadsoft.com
20.4.4 Database Synchronization Monitoring

The Application Server, Network Server, and Messaging Server peer databases are kept
synchronized using TimesTen replication. TimesTen replication is not a Database
mirroring application, but instead pushes transactions from one server to the other. If the
replication daemon is stopped on either peer, the peer databases may not be
synchronized.
Database synchronization can be verified manually by running the synchcheck_basic.pl
script on the secondary server using the –a option.
bwadmin@bsns2$ synchcheck_basic.pl -a
Report:
synchcheck_basic.pl started Fri Jul 29 10:31:35 EDT 2005
Performing basic comparison between datastore NetworkServer on bsns2
(local) and NetworkServer on ns1
Databases appear in sync
Recommended Actions:
None
The synchcheck_basic.pl performs a row count comparison of every database table. It is

not service affecting and can be run at any time. If discrepancies are discovered, it
indicates the TimesTen table in issue.
The dbSyncCheck is scheduled task that should be configured on the secondary server to
run every night. It performs a database comparison and sends a critical severity
bwDatabaseSyncReport trap if any discrepancies are discovered.
NOTE: The synchcheck_full.pl script performs a row-by-row comparison of the peer databases.
This script severely affects performance in that it locks both peer databases during the compare.
In addition, synchcheck_full.pl requires twice the amount of the database size memory free on
the server running the script, since the peer database is imported and compared in memory. It is
not recommended to run synchcheck_full.pl on a production server.

©
21 BroadWorks Application Server Time Zone Support
The valid time zones supported by the BroadWorks Application Servers are stored in the
following configuration files:
 TimeZoneAlias.conf
 TimeZoneAliasLabels.conf
TimeZoneAlias.conf contains a list of valid time zones. Each line has a time zone ID
display name and a supported flag. If the supported flag is set to “Y”, then the time zone
can be used by users and groups. All time zones used in BroadWorks should be included
in the TimeZoneAlias.conf file. A system default time zone can be configured for the
Application Server that is running BroadWorks. The default time zone is also a supported
time zone even if it is not included in the TimeZoneAlias.conf file.
TimeZoneAliasLabels.conf contains a list of labels for those supported time zones in the
TimeZoneAlias.conf file. Each line contains two values; the first one is the time zone
display name, which has to match the display name value in the TimeZoneAlias.conf file.
The second value is the label used to display the time zones. For more information on
how to localize this file for multiple languages, refer to the BroadWorks Web Portal
Customization and Localization Guide.
The list of supported time zones are returned by the
SystemTimeZoneGetListRequest OCI-P command, and can be viewed on the web
portal as a drop-down list wherever the time zone can be assigned. This includes the
Group Profile, User Profile, and Call Center Profile pages.
When a time zone is assigned to a user or group for example, it is validated against the list
of supported time zones as mentioned above. However, after a time zone is assigned, it
can become invalid after one (but not limited to) of the following changes:
 The time zone is removed from the TimeZoneAlias.conf file or not “supported” any
longer.
 The system default time zone is not in the TimeZoneAlias.conf and has been
changed.
 The time zone name is changed in java following a system/java upgrade.
If a group has an invalid time assigned, the system default time zone is returned as part of
the group profile, when requested. If a user (real user or virtual user such as call center or
hunt group) has an invalid time zone, the group time zone (or the system default time zone
if the group time zone is invalid) is returned as part of the user profile.
When adding or modifying a user or a group, only the valid time zones can be assigned or
modified.

©
22 BroadWorks Scripts and Utilities
22.1 bwBackup.pl
Usage – bwBackup.pl <dsn> <file>
NOTE: A database backup can also be initiated from the CLI /Maintenance/Tools level using the
backupdb command.
Description – This script backs up the content of the BroadWorks TimesTen datastore to
the specified directory. Database backups can be initiated with the application running.
22.2 bwCPUMon
Usage – bwCPUMon [-h]
Description – CPU monitoring script (for CPU idle percentage, and memory usage) that
can be used to automatically generate a trap with severity level depending on the
configured idle time threshold that has been surpassed. This script should be set up via
the CLI as a configurable maintenance task at five-minute intervals.
The generated BroadWorks traps are:
 bwCPUIdleTimeLimitReached (multiple thresholds)
 bwMemoryLimitReached (one threshold)
22.3 bwListPortInUse
Usage – bwListPortInUse
Description – This tool provides a list of all ports transmission control protocol (TCP) and
user datagram protocol (UDP) configured to be used by BroadWorks.
Sample of bwListPortInUse:
Usage Type Port Internal
-------------------------------------------------------------------
SNMP agent UDP 8001 N
SNMP agent trap server TCP 30002 Y
MO daemon TCP 7120 Y
MO root daemon TCP 7121 Y
SW Manager TCP 2223 N
SW Manager (Perl) TCP 2224 Y
TimesTen Main Daemon TCP 17003 N
XLA Server TCP 2048 Y
Replication port TCP 17888 N
ASR interface UDP 5090 N
CAP Server TCP 2206 N
MGCP interface UDP 2727 N
SMDI server UDP 11234 N
SIP listening port UDP 5060 N

©
22.4 bwRestore.pl
Usage – bwRestore.pl <dsn> <file>
NOTE: A database restore can also be initiated from the CLI /Maintenance/Tools level using the
restoredb command.
Description – This script restores the content of the BroadWorks TimesTen datastore
from the specified directory. The content must have previously been backed up with
bwBackup.pl.
There can be no connection to the database when restoring a database (the application
must be stopped and replication must also be stopped). Logs for the database restore
can be found in /var/broadworks/restore.log.
22.5 bwshowver
Usage – bwshowver
Description – This script is used to view the active BroadWorks release and applied
patches. For a list of applied patches, use the bwshowver –showPatch option.
Sample of bwshowver:
bwadmin@viras01.mtl.broadsoft.com$ bwshowver
AS version Rel_17.0_1.430
Built Fri Mar 26 09:15:30 EDT 2010

- BASE revision 139067
- AS revision 139067
Patching Info:
Active Patches: 0
22.6 bwTestDaemon.pl
Usage – bwTestDaemon.pl [-h]
Description – This command should be used to validate that the MoDaemon and
MoDaemonRoot are properly configured and operational on the target server.
Sample of bwTestDeamon.pl:
bwadmin@MTLAS04$ bwTestDaemon.pl
************************************************************
* BroadWorks bwTestDaemon.pl *
* *
* Version: 1.0 *
************************************************************
Validating data
Validating configuration
Validating bwMoDaemon
... Found port value: 7120
Validating bwMoDaemonRoot
... Found port value: 7121

©
Validating connectivity
Validating bwMoDaemon on host:localhost and port:7120
Validating bwMoDaemonRoot on host:localhost and port:7121
Done
22.7 check_dbpages.pl
Usage – check_dbpages.pl <dsn> <detect|modify>
Description – This utility is used to optimize the paging in the database. The detect
option can be used to check if any table is either undersized or oversized, and the modify
command is used to perform the optimization.
NOTE: This script should be executed during a maintenance window as it causes the database
to be temporarily locked.
bwadmin@mtl64lin02$ check_dbpages.pl networkserver detect

bwadmin@mtl64lin02$ check_dbpages.pl networkserver modify
************WARNING************
This script should only be run during a maintenance window!

This script will cause the database to lock temporarily.
Do you want to continue (y/n)? y
ttIsql (c) 1996-2005, TimesTen, Inc. All rights reserved.

Type ? or "help" for help, type "exit" to quit ttIsql.
All commands must end with a semicolon character.
connect "dsn=networkserver";
Connection successful:
DSN=NetworkServer;UID=bwadmin;DataStore=/usr/local/broadw
orks/bw_base/persistent/NetworkServer;AutoCreate=0;LogFileSize=8;DurableC
ommits=
0;WaitForConnect=0;DRIVER=/usr/local/TimesTen/tt_base/lib/libtten.so;SMPO
ptLevel
=1;LogBuffSize=8192;Authenticate=0;PermSize=128;TempSize=42;
(Default setting AutoCommit=1)
run /tmp/tmp_file.sql;
exit;
Disconnecting...
Done.

©
22.8 config-hostname
Usage – config-hostname
Description – This utility is used whenever a host name must be changed for a
BroadWorks server. This utility calls the configure-network script.
Sample of config-hostname:
---- Hostname Configurator version 1.0 ----
Data Gathering 
What is the new hostname? [as1]^C
as1$ config-hostname ?
---- Hostname Configurator version 1.0 ----
Data Gathering 
What is the new hostname? [as1]
What is the new address? [as1.lab.broadsoft.com]
Changing system hostname 
Please enter root password:
Setting new hostname to as1… [done]
now processing /etc/hosts… [done]
now processing /etc/nodename… [done]
now processing /etc/net/ticlts/hosts… [done]
now processing /etc/net/ticots/hosts… [done]
now processing /etc/net/ticotsord/hosts… [done]
now processing /etc/hostname.dmfe0… [done]
You may have to manually update the HTTP interfaces

(CLI/Interface/Http)
==> BroadWorks configuration <==

Configuring new BroadWorks hostname... [done]
Restart replication on all servers of the cluster using repctl
restart.
---- Hostname Configurator completed! ----
22.9 config-redundancy
Usage – config-redundancy
Description – This utility is used to add a peer to a cluster. It calls the config-ssh utility.
22.10 config-ssh
Usage –./config-ssh [-createKeys | -remote] peer1 peer2 …
Description – This is an SSH configuration utility that is used to create keys, propagate
them to other peers, and ensure connectivity between the peers.
Options:
 createKeys allows SSH keys to be created for each peer
 remote requests that peers do not iterate down to other peers
peer1 peer2 … are the FQDN (equal to host name) of all peers.

©
22.11 dbsetup.sh
Usage – dbsetup.sh <bw_version> [-install]
WARNING: The dbsetup script is wrapped by other BroadWorks scripts and should not be run
unless requested by BroadSoft Support.
Description – This script encompasses all aspects of database setup. It may perform
one or more of the following steps depending on the desired configuration:
 Backs up the datastore to upgrade.
 Installs a fresh datastore, meaning that tables are created and basic data fill is added
(using the -install option).
 Populates the list of redundant peers.
 Locks all redundant servers before performing an upgrade.
 Unlocks the current server once an install or upgrade is complete.
 Upgrades a previous datastore (the backup version is restored in the new location
and patches are applied).
It uses directly or indirectly the following scripts:
 applyPatches.sh
 dbpatch.sh
 peerctl
 bwBackup.pl
 bwRestore.pl
 applyPatches.sh
22.12 getDbSize.pl
Usage – getDbSize.pl <dsn>
Where – dsn - Datastore name (either CallCenterRDB, radius, or WebNmsDB)
Description – This tool provides the MySQL database size of the given datastore name.
It is available for the Element Management System.
22.13 healthmon
Usage – healthmon <-h|-v|-s|-l|-c> [-p] [-d] [5,10,15,20,30,60,120,720,1440,remove,show]
Options – Options are as follows:
 -h: print this usage note
 -v: print the version of the script
 -s: generate a report and send the result through an SNMP trap
 -l: generate a local to screen
 -p: test only the disk partitions status
 -d: generate a detailed trace of all the tests executed

©
This option is ignored for an SNMP report.
 -c: schedule the script to run in background with the -s option
You must specify a valid scheduling timeslot:
− Every 5, 10, 15, 20, or 30 minutes
− Every hour (60), every other hour (120), twice a day, (720) or daily
 Keyword “remove” removes the healthmon scheduling entry
 Keyword “show” shows the current schedule
Description – This tool is used to monitor and report the status of a BroadWorks server.
The report can be generated locally at the console or sent over the network through an
SNMP trap. Note that a trap is still sent even when no problems are detected (keep alive).
22.14 importdb.pl
Usage – importdb.pl <local dsn> <remote hostname> <remote dsn>
Where:
 local dsn – Local datastore name (either AppServer or NetworkServer)
 remote host name – Remote server UNIX host name
 remote dsn – Remote datastore name (either AppServer or NetworkServer)
NOTE: A database import can also be initiated from the CLI Maintenance/Tools level using the
importdb command.
Description – This script imports the content of a remote database in the specified
datastore. The remote server must have its replication enabled. Logs for the database
import can be found in /var/broadworks/importdb.log.
22.14.1 importdb.pl on Element Management System

On the Element Management System server, the importdb.pl command is defined
differently compared to how it is on other BroadWorks servers.
Usage – importdb.pl <remote hostname>
Where:
 remote host name – Remote server UNIX host name
Example:
bwadmin@ems2$ importdb.pl ems1
Re-spawning as root...
-> Disabling peer database replication <-

[done]

Backing up ems local db to
/var/broadworks/backup/EMS_Rel_14.sp3_1.200.081114120132.backup...
[done]
WARNING: ALL information in ems db will be lost.

©
Press ctrl-c now to abort or <enter> to continue...
Continuing.
-> Backing-up data on primary server <-

[done]
[done]
-> Destroying ems database <-
[done]
-> Extracting data to ems database <-
[done]
-> Re-setting local redundancy data <-
[done]
-> Re-setting peer redundancy data <-
[done]
WARNING: You should re-enable local database replication.
-> Re-enabling peer database replication <-
[done]
[Execution Completed]
bwadmin@ems2$
22.15 installdialplan.sh
Usage – ./installdialplan.sh <NS version> [--run-once]
Description – This script is used to install new dial plans on the Network Server. Dial
plan files can be created offline and loaded by placing the new dial plan in the
/usr/local/broadworks/bw_base/persistent/setup/dataoptional directory and updating the
choice.lst with the new entry.
Sample of $ ./installdialplan.sh NS_Rel_8.0_1.43:
==> Dial plan installation <==
The following dial plans are already installed:
CC Name
--------------
1 NADP
1 NADP_E164
1 NADPCAC
Optional dial plan list:

CC FILENAME Description
------------------------------------------------
[1] NS.CC_DIAL_PLANS "North American dial plan"
[61] NS.AUSTRALIA_DIAL_PLANS "Australian dial plan"
[65] NS.SINGAPORE_DIAL_PLANS "Singapore dial plan"
[81] NS.JAPAN_DIAL_PLANS "Japan dial plan"
Which dial plan would you like to install (enter number only)? 65
Now installing NS.SINGAPORE_DIAL_PLANS...
==> Dial plan installation completed! <==

©
22.16 peerctl
Usage – peerctl [command]
Where the command must be one of the following:
 ls
 add <hostname> <address> <repType(0=master;1=slave)>
 set <hostname> <address> <repType(0=master;1=slave)>
 setReplicationAddress <hostname> <replication routing address>
 clearReplicationAddress <hostname>
 showReplicationRouting
 del <hostname>
 lock [<hostname>|all]
 unlock [<hostname>|all]
 setPrimary [<hostname>]
 waitAllLocked <timeout in minutes (typically 12)>
Where host name is optional (omitting it means the current peer).
Description – This script is used to manage cluster peers. Its primary use is to list all
cluster peers, add/delete peers from a cluster, and lock/unlock the TimesTen database. It
can also be used to specify a different interface for TimesTen replication than the
configured peer address.
The following example illustrates how to set an address of the replication interface.
bwadmin@mtlsol25$ peerctl setReplicationAddress MTLSOL-25 192.168.13.172
Replication traffic for peer 'MTLSOL-25' will be routed via address

'192.168.13.172'
BroadWorks replication must be restarted by using the 'repctl' command
before this change will take effect.
To ensure that replication can function properly, you must generate ssh
keys for
the new routing address '192.168.13.172' by using the 'config-ssh -
createKeys' utility
As indicated, ssh keys, for the new interface must be generated as follows.
bwadmin@mtlsol25$ config-ssh -createKeys bwadmin@<new-replication-
address-on-Prim> bwadmin@<new-replication-address-on-Secondary>
If you want to clear the routing address and put back the initial settings, execute the
following command.
bwadmin@mtlsol25$ peerctl clearReplicationAddress MTLSOL-25
WARNING: When the cluster is locked through the peerctl lock all command, the device
registrations and provisioning changes are prevented.

©
22.17 peercmd
Usage – peercmd <command to spawn across BroadWorks peers>
Description – This utility spawns a given command to all peers as listed by “peerctl ls”. It
can be used to simultaneously launch an action or gather the status on a BroadWorks
cluster. Complex commands can be created with escaped control characters.
Example: $ peercmd bwshowver\;showrun\&repctl status
22.18 repctl
Usage – /usr/local/broadworks/bw_base/bin/repctl {start|stop|restart|status}
Description – This script is used to start, stop, and get the status of file and database
replication. This script should always be used to start and stop replication as it performs a
cleanup. SSH must be properly configured before starting replication. Failing to configure
SSH leads to broken file replication. A lock file is created in /var/broadworks/lock/repctl to
prevent multiple instances from running simultaneously.
Sample of status when replication is started:
bwadmin@mtl64lin10.mtl.broadsoft.com$ repctl status
-----------------------------
Datastore name = NetworkServer
Replication State
mtl64lin10.mtl.broadsoft.com: (filerep: started) (database: started)
mtl64lin17.mtl.broadsoft.com: (filerep: started) (database: started)

mtl64lin10.mtl.broadsoft.com: ok (1 files held)(earliest LSN 4/5321312)
mtl64lin17.mtl.broadsoft.com: ok (1 files held)(earliest LSN 3/6284464)

mtl64lin10.mtl.broadsoft.com: ok (1 files held)
(NetworkServerExecutionServer earliest LSN 4/5323240)
(NetworkServerProvisioningSer earliest LSN 4/5325672)
(CAPProxyServrUtility earliest LSN 4/5319664)
Sample of status when replication is not running:

bwadmin@mtl64lin10.mtl.broadsoft.com$ repctl status
-----------------------------
File Replication pid(s) =
Datastore name = NetworkServer
Replication Agent Policy : manual
Replication Manually Started : False
Replication State
mtl64lin10.mtl.broadsoft.com: (filerep: stopped) (database: stopped)
mtl64lin17.mtl.broadsoft.com: (filerep: stopped) (database: stopped)

©

22.19 resizeDSN
Usage – resizeDSN
WARNING: The resizeDSN tool requires the application to be stopped. This tool should be
used with caution and only used following an explicit acknowledgement from BroadSoft Support
personnel.
Description – This command can be used to resize the Application Server, Network
Server, or Messaging Server DSN. This tool can only be run as root.
NOTE: For information on database sizing, see the BroadWorks System Engineering Guide.
22.20 showrun
Usage – showrun
Description – This script is used to view all active BroadWorks processes. An empty list
means the application is not running.
Example of showrun:
bwadmin@MTLAS01$ showrun
Currently running (BroadWorks related) processes:
Execution Server process monitor (pid=2329)

Execution Server (pid=2367)
Provisioning Server process monitor (pid=2344)
Provisioning Server (pid=2387)
Open Client Server process monitor (pid=2391)
Open Client Server (pid=2429)
BroadWorks RemoteXla Server process monitor (pid=2290)

BroadWorks RemoteXla Server (pid=2322)
tnameserver (pid=2296)
Tomcat process monitor (pid=2372)

Tomcat (pid=2422)

©
Apache (pid=2438)
22.21 startbw
Usage – startbw
Description – This script is used to start the BroadWorks application. As part of the
startup sequence, this script also checks if database replication is lagging on any peers in
the cluster.
Example of startbw:
bwadmin@mtl64lin12.mtl.broadsoft.com$ startbw
Starting BroadWorks...
Broadcast message from bworks (Mon Apr 5 08:13:05 2010):
===== BROADWORKS CONTROL --- START INITIATED =====

Starting remotexla...
Starting nsExecution...
Starting nsProvisioning...
Starting tomcat...
Starting apache...
22.22 frepctl
Usage – frepctl <-start|-stop|-restart> [-daemon] [-once] [peer list] [-i, --interval timeInSec,
default 600] [-v, --verbose] [-h, --help]
WARNING: The rep.conf file should only be modified after consulting BroadSoft Support
personnel. Adding new directories to file replication may affect system performance. This script
is called from the repctl script and under normal conditions, should not be run manually.
Description – This command is used to start RSYNC file replication. This command
starts ssh-agent, holds SSH passphrase, and iterates over the list of configured peers to
synchronize configured files. Files or directories can be added or removed from file
synchronization in /usr/local/broadworks/bw_base/conf/rep.conf. In rep.conf, comments
are supported (comments start with a #) and entries can be a specific file or a directory.
For a directory, you must add a “.” to clearly specify that it is the content that is replicated.
Logs reside in /var/broadworks/logs/rsync/frepXX.log, where XX is the current day. A lock
file is created in /var/broadworks/lock/frepctl.pid to prevent multiple instances from running
simultaneously.
Example conf/rep.conf:
#File replication list
#[2001-22-13 Basic file replication configuration]
/bw/local/broadworks/bw_base/public_html/.
/bw/local/broadworks/bw_base/conf/DebugConfig.xml

©
22.23 stopbw
Usage – stopbw [--force]
Description – This script is used to gracefully stop the BroadWorks application. The --
force option to force BroadWorks to stop even if a service disruption will occur (for
redundant servers only).
WARNING 1: The following error can occur:
BroadWorks cannot be stopped at this time…
This occurs on redundant servers if a disruption of service is expected after BroadWorks is

stopped (peer is not available for servicing). This check can be bypassed by using the –force
option.
WARNING 2: The following error can occur:
***FAILURE***
***Some processes did not terminate gracefully
***Killing -9 (may result in broken pipes)…
This occurs because some processes take more time to shut down. If the showrun command
does not give any running process after the stopbw, everything is fine.
Example stopbw:
bwadmin@mtl64lin12.mtl.broadsoft.com$ stopbw
Broadcast message from bworks (Mon Apr 5 08:12:02 2010):

Stopping tomcat...
Stopping apache...
Waiting for core processes to terminate.........
... Done.
Waiting for core processes to terminate.......
... Done.


©
22.24 synchcheck_basic.pl
Usage – synchcheck_basic.pl [-s] <-a | <local dsn> <remote hostname> <remote dsn>
Description – This script is used for redundant Application and Network Servers, to make
sure that the specified local database is identical to remote databases.
This command is used to determine if two databases are synchronized. It performs a
basic synchronization sanity test, which validates that peer databases are synchronized. It
checks that the number of rows in database tables is the same on peer servers, and
provides a list of suspect tables on mismatches.
22.25 synchcheck_full.pl
Usage – synchcheck_full.pl
WARNING: This script can take ten minutes or longer and keeps servers locked while running.
Description – This script for redundant Application and Network Servers ensures that the
specified local database is identical to the remote databases.
All local and remote servers are locked while the script runs, so it should only be run in
periods of low system usage. No changes to the databases take effect while the script is
running.
22.26 ttStatus
Usage – ttStatus
Description – This tool provides information on the TimesTen databases in the system.
Connections to the database indicate that the database is in use. These connections are
related to the TimesTen replication daemon, the BroadWorks application, or a manual
connection via ttIsql.
Example ttStatus:
bwadmin@viras01.mtl.broadsoft.com$ ttStatus
TimesTen status report as of Sun Oct 16 10:23:15 2011
Daemon pid 1842 port 11284 instance 11.2.1.8.4

TimesTen server pid 1850 started on port 11285
------------------------------------------------------------------------
Data store /bw/broadworks/persistent/AppServer
There are 48 connections to the data store
Shared Memory KEY 0x10036160 ID 557061
Type PID Context Connection Name ConnID
Process 3087 0x000000001821aa90 remotexla 5
Process 3087 0x0000000018323440 remotexla 6
Process 3583 0x00000000468846d0 Execution Server 7
Process 3583 0x00000000469f81d0 Execution Server 12
Process 3583 0x0000000046a6e450 Execution Server 13
Process 3583 0x0000000046ad5040 Execution Server 14
Process 3583 0x0000000046b3fa50 Execution Server 15
Process 3583 0x0000000046be03a0 Execution Server 10
Process 3583 0x0000000046d96c30 Execution Server 18
Process 3583 0x000000004710a1b0 Execution Server 20
Process 3583 0x0000000047175190 Execution Server 23
Process 3583 0x0000000047225e10 Execution Server 24

©
Process 3583 0x00000000474ae430 Execution Server 34
Process 3583 0x00002aaab73185e0 Execution Server 33
Process 3583 0x00002aaab737cdb0 Execution Server 29
Process 3583 0x00002aaab75e85b0 Execution Server 31
Process 3583 0x00002aaab76a4bd0 Execution Server 32
Process 3583 0x00002aaab7c20d00 Execution Server 36
Process 3583 0x00002aaab7cbad70 Execution Server 30
Process 3583 0x00002aaab7e4d260 Execution Server 38
Process 3583 0x00002aaab84b8b70 Execution Server 27
Process 3583 0x00002aaab8703420 Execution Server 25
Process 3583 0x00002aaab8b78290 Execution Server 28
Process 3583 0x00002aaabc2cc340 Execution Server 26
Process 3583 0x00002aaabc45d840 Execution Server 8
Process 3583 0x00002aaabc6cd340 Execution Server 35
Process 3611 0x0000000045957cf0 Provisioning Server 22
Process 3611 0x00000000459a2170 Provisioning Server 21
Process 3611 0x00002aaab42abaa0 Provisioning Server 16
Process 3611 0x00002aaab443aef0 Provisioning Server 9
Process 3611 0x00002aaab455f620 Provisioning Server 17
Replication 19582 0x000000001d0e9a60 LOGFORCE 4
Replication 19582 0x000000001d109e70 REPHOLD 2
Replication 19582 0x000000001d15d4c0 REPLISTENER 3
Replication 19582 0x000000001d1b0b10 TRANSMITTER 37
Replication 19582 0x000000001d2285b0 RECEIVER 1
Subdaemon 1848 0x00000000152d2b50 Manager 2032
Subdaemon 1848 0x000000001533e750 Flusher 2034
Subdaemon 1848 0x0000000015391fc0 Monitor 2035
Subdaemon 1848 0x00000000153e5830 Deadlock Detector 2036
Subdaemon 1848 0x00000000154390a0 Checkpoint 2037
Subdaemon 1848 0x000000001548c910 Aging 2038
Subdaemon 1848 0x00000000154e0180 Log Marker 2039
Subdaemon 1848 0x00000000155339f0 AsyncMV 2040
Subdaemon 1848 0x0000000015587260 HistGC 2041
Subdaemon 1848 0x00002aaaac042b70 Rollback 2033
RAM residence policy: Always
Replication policy : Always
Replication agent is running.
Cache Agent policy : Manual
------------------------------------------------------------------------
Accessible by group bwadmin
End of report
22.27 bwLoadHardening.pl
Usage – bwLoadHardening.pl
Description – This tool is used to modify the system configuration with the BroadSoft
hardening recommendations as defined in the BroadWorks System Configuration Guide.

©
22.28 Software Manager Utilities
22.28.1 backup-swmanager.pl
Usage – backup-swmanager.pl [-h | -help | <filename>]
Description – This script is used to back up the Software Manager database. The
backup file is stored in /usr/local/broadworks/swmanager/dbbackup. This script is located
in /usr/local/broadworks/swmanager/sbin.
NOTE: Backing up the patch tool database can cause a loss of information and it should be
done only when the patch tool is stopped.
22.28.2 restore-swmanager.pl
Usage – restore-swmanager.pl [<filename>]
Description – This script is used to restore a Software Manager database file created by
the backup-patchtool.pl script. The backup file includes the Software Manager database
and it should only be used with the same Software Manager revision that was used to
back it up. The backup-swmanager.pl script includes the Software Manager revision
number in the name of the backup file. This script is located in
/usr/local/broadworks/swmanager/sbin.
22.29 configdctl
Usage – configdctl {start | stop | stopnow | restart | status}
Description – This script is used to control the configuration agent daemon.
This command is used to start, stop, restart, and get the status of the configuration agent
daemon.
NOTE: The configuration agent must not be stopped while BroadWorks is running. This can
lead to unexpected behavior.

©
23 Miscellaneous Maintenance Tasks
23.1 Add Memory to BroadWorks Server

At installation time, BroadWorks optimizes its configuration based on server
characteristics. For instance, memory usage of the different processes is calculated and
validated against the available amount of RAM. To take advantage of new memory
installed on a server, an administrator must perform the following:
 Access the CLI and navigate to System/ProfileTuning/GeneralSetting.
 Use the “get” command to determine which profile is currently used for the server.
 Use the “clear” and “set” commands to clear and set the profileTuningName to the
desired profile to retune the server according to the new hardware capabilities.
 Restart BroadWorks.
In the event that the EMS is used to manage the server configuration, the node must be
refreshed once its hardware has been updated. The server’s configuration can then be
edited to reapply the server profile.
23.2 Adjust BroadWorks Daylight Savings Time
23.2.1 Description
In 2005 and 2006, Daylight Saving Time (DST) began for most of the United States at
2 A.M. on the first Sunday of April. Then time reverted to standard time at 2 A.M. on the
last Sunday of October.
United States and Canada are extending Daylight Saving Time from 2007, and other
regions and countries in North America may follow. The new start date is the second
Sunday in March (currently the first Sunday in April) and the new end date is the first
Sunday in November (currently the last Sunday in October).
Year DST Begins 2 a.m. DST Ends 2 a.m.
First Sunday in April) (Last Sunday in October)
==== ========================== ===============
2005 April 3 October 30
2006 April 2 October 29
NOTE: The DST changes to the start and end date began March 2007.
Year DST Begins 2 a.m. DST Ends 2 a.m.

(Second Sunday in March) (First Sunday in November)
==== ========================== =============================
2007 March 11 November 4

©
The new start and stop dates were set in the Energy Policy Act of 2005 in the United
States. In addition, the majority of the Canadian provinces have announced that they are
changing the dates for the DST to align with the U.S. Only the province of Saskatchewan,
which does not observe DST, as well as Nunavut Territory are not changing. Note that
Western Australia has introduced DST for a three-year trial period that started December
3, 2006 and ended March 25, 2007.
23.2.2 Service Impact

DST of current users on Application Servers is “out-of-phase” with the new DST as
described by the Energy Policy Act of 2005. This impacts time-oriented services, call log
date/time stamps, log message header time stamps, and so on. This alert needs to be
implemented prior to March 11, 2007.
Although not essential for the Xtended Services Platform, Network Server, and Media
Server since there is no time zone used on these servers, it is still recommended to
proceed with the following instructions. This is because the XSlogs, ms.syslog, and the
snmp traps have time stamps associated with them, which means that logs and traps may
not be synchronized with other servers.
23.2.3 Solution
Solaris SPARC:
Solaris 8 with time zone patch 109809-11 or later, and libc patch 108993-52 or later
Solaris x86:
NOTE: For more information, see Sun Alert 102775.
Linux:
Red Hat Enterprise Linux 4 – tzdata-2005m-1.EL4 is required. Connect to Red Hat Linux
and download this via the up2date command. For more information, read
http://kbase.redhat.com/faq/docs/DOC-4368.
After using the up2date command, users must set the /bin/nice setuid bit: # chmod ug+s
/bin/nice. After running the chmod command as root, restart BroadWorks (reference Alert
222646).
BroadWorks (on Solaris and Linux):

Release 11.1 – All servers: Execute the tzupdater tool.
Release 13.0 – Application Server only: Upgrade the JDK to JDK1.5.0_07 from 1.5.0_03.
After the upgrade, execute the tzupdater tool.

©
23.2.4 Instructions
tzupdater Tool
NOTE: The official Sun “read me” is http://java.sun.com/javase/tzupdater_README.html.

Download the tool from http://java.sun.com/javase/downloads/index.jsp (JDK US DST Timezone
Update Tool – 1.0.1).
Perform the following in a maintenance window as bwadmin, on each BroadWorks server

node:
1) Make sure that peer BroadWorks servers are updated before moving on to another
BroadWorks server type.
2) Download the tool to each BroadWorks server.
3) Create directory /export/home/bwadmin/DSTChange.
4) Move the update tool archive into this directory.
5) Extract the archive within this directory.
6) Lock BroadWorks from the command line interface (CLI). Wait until the server is
locked.
7) When the server is locked, exit the CLI and stop BroadWorks.
8) Navigate to the /usr/local/java/java_base/bin directory.
9) Execute /java -jar <full path of location>/tzupdater.jar –u.
23.3 Change Media Server Kernel Selection on Linux

In some cases, it may be required to select a previously installed kernel to recover from
one of the following conditions:
 The newer kernel installed during a Media Server upgrade causes kernel panic.
 The kernel was upgraded to a more recent kernel than the one included in
BroadWorks packages and no longer has the real-time extensions.
To recover from one of these conditions, these steps must be followed:
1) From the server’s console, reboot the faulty server.
2) While the server boots, the GNU GRUB window appears. There is a five-second
count down. Press the ESCAPE key (ESC).
3) Using the down arrow, navigate to the previous kernel. (It should be “Red Hat
Enterprise Linux WS (2.6.9-42.0.2.EL.bworkssmp)”).
4) Press ENTER.
5) Once the server has booted, log in as root.
6) As root, remove the faulty kernel, for example, as root run “rpm –e kernel-smp-2.6.9-
89.0.11.EL.bworks”.
7) As root, edit /boot/grub/menu.lst and verify that the default is set to the zero-based
index of the working BroadWorks kernel.

©
Example:
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this
file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux WS (2.6.9-42.0.2.EL.bworkssmp)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.0.2.EL.bworkssmp ro
root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.9-42.0.2.EL.bworkssmp.img
title Red Hat Enterprise Linux WS (2.6.9-42.0.2.EL)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.0.2.EL ro root=/dev/VolGroup00/LogVol00
rhgb quiet
initrd /initrd-2.6.9-42.0.2.EL.img
8) Reboot the server.

©
Index
Activation, Application management, 73 Application Server migrated user monitoring, 233

Active software versions, 88 Application Server, Execution Server, Profile Server
Display full details, 88 BroadWorks performance measurements
Software version management, 89 monitoring, 205
Add Application Server, Network Server, Messaging
Memory to BroadWorks server, maintenance Server
tasks, 251 Add new peer to redundant system, 124
New peer to redundant system Database replication, Redundancy
Application Server, Network Server, management, 113
Messaging Server, 124 Application state, 71
Element Management System, 129 Server management, 68
Profile Server, 131
Attributes, threshold alarms, 60
Redundancy management, 124
Automatic backups
Additional MySQL maintenance tasks
File naming conventions, 94
Database, 111
Mode of operation, 93
Start and stop MySQL database daemon, 111,
Restoring auto-backup, 94
112
Scheduled maintenance tasks, 93
Update MySQL database statistics, 111
Automatic disk cleanup, scheduled maintenance
Additional TimesTen maintenance tasks
Change TimesTen database size, 106 tasks, 96
Compact database, 108 Background, Solaris DiskSuite maintenance, 175
Database, 106 Backup
Import TimesTen datastores between operating Database, 100
systems, 108 Database maintenance, 99
Start and stop database daemon, 107 Manual database, 99
TimesTen connection optimization, 110 backup-swmanager.pl, 250
Update TimesTen database paging information, BCCT listening ports, changing, 174
107
BroadWorks Application Server, time zone support,
Update TimesTen database statistics, 107
235
Alarms
Clear, 60 BroadWorks license files, installing, 155
Close system, 60 BroadWorks performance measurements monitoring
Enable and disable real-time alarms echoing, Application Server, Execution Server, Profile
59 Server, 205
Fault and performance measurements, 56 Media Server, 193
Open system, 56 Network Server, 200
View and set CLI alarm backlog size, 58 BroadWorks server restoration, 141
View system, 57 BroadWorks system backup, 141
Apache logs, 55 License, 142
Application management, 71 Procedure, 142
Activation, 73 BroadWorks system backup, 141
Deactivation, 76 BroadWorks system monitoring, 185
Deployment, 74 Application Server migrated user, 233
Installation, 72 Database synchronization, 234
Undeployment, 74 Performance measurements, 193
Uninstallation, 76 Platform, 185
Application Server and Network Server Redundancy monitoring, 232
Database replication SNMP trap, 223
TimesTen database synchronization, 113 System sanity, 230
TimesTen replication monitoring, 115 BroadWorks versions, uninstalling obsolete, 156
TimesTen replication port number, 117 bwBackup.pl, scripts and utilities, 236
Application Server Execution Server bwCPUMon, scripts and utilities, 236
Heap monitoring, platform, 190, 191 bwListPortInUse, scripts and utilities, 236
Process problems, SNMP trap monitoring, 229 bwLoadHardening.pl, scripts and utilities, 249
Application Server Execution Server, Profile Server bwRestore.pl, scripts and utilities, 237
traps, SNMP trap monitoring, 226

©
bwshowver, scripts and utilities, 237 configdctl, 250
bwTestDaemon.pl, scripts and utilities, 237 config-hostname, scripts and utilities, 239
Change config-redundancy, scripts and utilities, 239
BCCT listening ports, 174 config-ssh, scripts and utilities, 239
HTTP server network configuration, updating Configure
BroadWorks network configuration, 164 IP networking information at UNIX level,
TimesTen database size, additional TimesTen changing IP address or host name, 162
maintenance tasks, 106 Redundant routing, changing IP address or host
Change IP address or host name name, 165
Configure IP networking information at UNIX SSH between servers, SSH/RSYNC
level, 162 configuration, 135
Configure redundant routing, 165 Threshold alarms, CLI, 61
Import EMS database, 173 Thresholds, 63
Update Application Server and Network Server Cookie password, 154
address on OCS, 170
CPU
Update Application Server call processing
Monitoring tool, scheduled maintenance tasks,
routing information, 165
98
Update Application Server IP address on
Usage, platform monitoring, 186
Network Server, 166
Update EMS redundancy configuration, 172 Create SSH keys, SSH/RSYNC configuration, 135
Update Media Server IP address on Application Database backup
Server, 170 Application Server, Element Management
Update Media Server IP address on Network System, Call Center Reporting Server,
Server, 169 Network Server, Messaging Server, 100
Update network configuration, 164 Maintenance, 99
Update Network Server IP address on Manual, 99
Application Server, 167 Database import
Update Network Server or Application Server Application Server, Element Management
address on Xtended Services Platform, System, Network Server, Messaging
172 Server, 103
Update Network Server routing alias list, 168 Maintenance, 102
Update Xtended Services Platform IP address Database installation, maintenance, 104
on Network Server, 171 Database maintenance, 99
Change IP address, host name, FQDN name, Additional MySQL maintenance tasks, 111
gateway, or DNS server, 158 Additional TimesTen tasks, 106
Changes Backup, 99
History, 21 Import, 102
Release 10.0, 30 Installation, 104
Release 11.1, 29 Recreate Application Server, Network Server,
Release 12.0, 29 or Messaging Server database, 104
Release 13.0, 29 Restore, 100
Release 14.0, 27 Database restore, 100
Release 15.0, 26 Application Server, Element Management
Release 16.0, 26 System, Call Center Reporting Server,
Release 17.0, 24 Network Server, Messaging Server, 101
Release 18.0, 23 Extract database backup from automatic
Release 19.0, 23 backup file, 101
Changing Media Server Kernel Selection on Linux, Maintenance, 100
maintenance tasks, 253 Database scheduled tasks, maintenance tasks, 98
Check for running process, server management, 79 Database size monitoring, platform, 192
check_dbpages.pl, scripts and utilities, 238 Database statistics, scheduled maintenance tasks,
Clear, alarms, 60 97
CLI alarm backlog size, 58 Database synchronization
Close system alarms, 60 Check, scheduled maintenance tasks, 98
Monitoring, 234
Common BroadWorks server traps, SNMP trap
dbsetup.sh, scripts and utilities, 240
monitoring, 223
Deactivation, Application management, 76
Compact database, additional TimesTen
Debug logs, 38
maintenance tasks, 108
Debugging

©
Enable or disable debug logging, 52 Import
Input channel attributes, 51 Database, 102, 103
Log input channels, 39 EMS database, changing IP address or host
Log rotation, 53 name, 173
Log severity, 39 TimesTen datastores between operating
Output channel attributes, 52 systems, additional TimesTen
Sample debug configuration file, 53, 54 maintenance tasks, 108
Deployment, Application management, 74 importdb.pl
Description, DST adjustments, 251 Element Management System, 241
Device operational state monitoring, SNMP trap, 229 Scripts and utilities, 241
Disable cookie support, modifying Web.xml file, 154 Input channel attributes, debugging, 51
Disable debug logging, 52 Install BroadWorks license files, 155
Disaster recovery strategy, 138 Install cluster of servers
BroadWorks server restoration, 141 Element Management System additional
Recover server OS, 138 parameters, 122
Disk activity, platform, 189 Redundancy management, 121
Disk cleanup, automatic, 96 Installation
Application management, 72
Display active software versions, software version
Database, 104
management, 88
installdialplan.sh, scripts and utilities, 242
DST adjustments, 251
Installed software versions, 88
Description, 251
List installed and active, 88
Instructions, 253
Service impact, 252 Instructions, DST adjustments, 253
Solution, 252 IP address, host name, FQDN name, gateway, or
Element Management System DNS server, changing, 158
Add new peer to redundant system, 129 License, 142
Additional parameters, installing cluster of License files, installing BroadWorks, 155
servers, 122 Limitation for Application Server thresholds, 64
Database replication
Linux system logs, 55
MySQL database synchronization, 119
MySQL replication monitoring, 120 List installed software versions, software version
Redundancy management, 119 management, 88
importdb.pl, 241 Lock Network Server or Application Server, force
Enable and disable real-time alarms echoing, 59 lock server, 80
Enable cookie support, modifying Web.xml file, 154 Lock Network Server, Application Server, or
Enable debug logging, 52 Execution Server
Extract database backup from automatic backup file, Server management, 80
restore, 101 Log input channels, debugging, 39
Fault and performance measurements Log rotation, debugging, 53
Alarms, 56 Log severity, debugging, 39
Performance, 65 Logs, 35
Threshold alarms, 60 Apache, 55
File system monitoring, platform, 185 Enabling TimesTen, 55
Force Linux system, 55
Lock server immediately, lock Network Server Miscellaneous, 55
or Application Server, 80 Oracle logs, 55
Server into maintenance state, server Solaris system, 55
maintenance state, 87 Tomcat logs, 55
frepctl, scripts and utilities, 246 UNIX command execution logging, 54
Get server state, 71 Maintenance
Database, 99
getDbSize.pl, scripts and utilities, 240
Recommendations, 31
Healthmon Solaris DiskSuite, 175
Keep alives, scheduled maintenance tasks, 97
Maintenance and recovery procedures
Scripts and utilities, 240
Solaris DiskSuite, 176
Server state monitoring tools, 84
System fails to boot, 179
History, changes, 21 System running, 176
Hot standby servers, 139 Maintenance tasks

©
Add memory to BroadWorks server, 251 Overload control monitoring, SNMP trap, 229
BroadWorks DST adjustments, 251 Password cookie, 154
Change Media Server Kernel Selection on
Peer, remove from redundant system, 133
Linux, 253
Miscellaneous, 251 peercmd, scripts and utilities, 244
Scheduled, 91 peerctl, scripts and utilities, 243
Management Performance measurements
Servers, 68 Fault and performance, 65
Software version, 88 Monitoring BroadWorks system, 193
Managing redundancy, 113 Navigate through MIB nodes, 65
View current position in MIB, 66
Manual database backup, 99
View MIB node values, 66
Measurements, fault, and performance, 56 View next MIB node, 65
Media Server Platform monitoring
BroadWorks performance measurements Application Server Execution Server heap, 190,
monitoring, 193 191
Traps, SNMP trap monitoring, 225 BroadWorks system, 185
Memory usage/paging monitoring, platform, 188 CPU usage, 186
Metadatabase replica status, Solaris DiskSuite, 176 Database size monitoring, 192
MIB node values, 66 Disk activity, 189
MIB nodes, 65 File system, 185
Memory usage/paging, 188
Miscellaneous monitoring, 230
Thread activity, 190
Mode of operation, automatic backups, 93
Post-installation control, Redundancy management,
Modify Web.xml file
134
Disable cookie support, 154
Enable cookie support, 154 Procedure, BroadWorks server restoration, 142
Remember password cookie for administrators, Processes, 32
154 Profile Server, adding new peer to redundant
MySQL system, 131
Database daemon, 111, 112 Real-time alarms echoing, 59
Database statistics, 111 Recommendations, maintenance, 31
Database synchronization, 119
Recover
Maintenance tasks, 111
Faulty redundant installation, Redundancy
Replication monitoring, 120
management, 134
Naming conventions, automatic backups, 94 Server in maintenance state, 87
Navigate through MIB nodes, performance Recover server OS, 138
measurements, 65 Hot standby servers, 139
Network Server Incremental backups, 140
BroadWorks performance measurements On-demand rebuild, 139
monitoring, 200 Recreate
Traps, SNMP trap monitoring, 225 Application Server, Network Server, or
Network Server and Application Server Messaging Server database, 104
synchronization, Redundancy management, Redundancy management, 113
124 Add new peer to redundant system, 124
Non-redundant systems Application Server Network Server, Messaging
Restore server to in-service, 82 Server database replication, 113
Take server out-of-service for maintenance, 81 Element Management System, database
replication, 119
Obsolete
Install cluster of servers, 121
BroadWorks versions, uninstalling, 156
Network Server and Application Server
Third-party components, uninstalling, 157
synchronization, 124
On-demand rebuild, 139 Post-installation control, 134
Open system alarms, 56 Recover faulty redundant installation, 134
Optional remember password cookie for Remove peer, 133
administrators, 154 SSH/RSYNC configuration, 135
Oracle logs, 55 Redundancy monitoring, BroadWorks system, 232
Out-of-service for maintenance, 81 Redundant routing, 165
Output channel attributes, debugging, 52 Redundant systems
Remove peer, 133

©
Restore server to in-service, 83 config-ssh, 239
Take server out-of-service for maintenance, 81 dbsetup.sh, 240
Registration audit, scheduled maintenance tasks, 97 frepctl, 246
Release 10.0, changes, 30 getDbSize.pl, 240
Healthmon, 240
Release 11.1, changes, 29
importdb.pl, 241
Release 12.0, changes, 29 installdialplan.sh, 242
Release 13.0, changes, 29 peercmd, 244
Release 14.0, changes, 27 peerctl, 243
Release 15.0, changes, 26 repctl, 244
Release 16.0, changes, 26 resizeDSN, 245
showrun, 245
Release 17.0, changes, 24
Software Manager, utilities, 250
Release 18.0, changes, 23 startbw, 246
Release 19.0, changes, 23 stopbw, 247
Remember password cookie for administrators, 154 synchcheck_basic.pl, 248
Modify Web.xml file, 154 synchcheck_full.pl, 248
repctl, scripts and utilities, 244 ttStatus, 248
Replacement disk requirements, Solaris DiskSuite, Server maintenance state
175 Force server into maintenance state, 87
resizeDSN, scripts and utilities, 245 Recover server in maintenance state, 87
Server management, 87
Restart configuration agent, updating BroadWorks
View current maintenance state, 87
network configuration, 164
Restart server, server management, 81 Application, 71
Restore Application state, 68, 71
Auto-backup, automatic backups, 94 Check for running process, 79
Database, 100, 101 Get server, 71
Database maintenance, 100 Lock Network Server, Application Server,
Restore server to in-service Execution Server, 80
Non-redundant systems, 82 Restart server, 81
Redundant systems, 83 Restore server to in-service, 82
Server management, 82 Server maintenance state, 87
restore-swmanager.pl, 250 Server state, 68
Running process, 79 Server state monitoring tools, 84
Start server, 76
Sample debug configuration file, debugging, 53, 54
Stop server, 77
Scheduled maintenance tasks, 91 Take BroadWorks server out-of-service for
Automatic backups, 93 maintenance, 81
Automatic disk cleanup, 96
Server process problems, SNMP trap monitoring,
Check DB pages, 98
CPU monitoring tool, 98 224
Database scheduled tasks, 98 Server state
Database statistics, 97 Get, 71
Database synchronization check, 98 Monitoring tools
Healthmon keep alives, 97 Healthmon, 84
Registration audit, 97 Server management, 84
Service license collect, 98 Tech-support, 86
Tech-support, 98 Server management, 68
Scripts and utilities, 236 Service impact, DST adjustments, 252
bwBackup.pl, 236 Service license collect, scheduled maintenance
bwCPUMon, 236 tasks, 98
bwListPortInUse, 236 showrun, scripts and utilities, 245
bwLoadHardening.pl, 249
SNMP trap monitoring
bwRestore.pl, 237
Application Server Execution Server process
bwshowver, 237
problems, 229
bwTestDaemon.pl, 237
Application Server, Execution Server, Profile
check_dbpages.pl, 238
Server traps, 226
configdctl, 250
BroadWorks system, 223
config-hostname, 239
Common BroadWorks server traps, 223
config-redundancy, 239
Device operational state, 229

©
Media Server traps, 225 System fails to boot, maintenance and recovery
Network Server traps, 225 procedures, 179
Overload control, 229
System monitoring, 185
Server process problems, 224
Miscellaneous, 230
Software error traps, 224
System running, maintenance and recovery
Software error traps, SNMP trap monitoring, 224
procedures, 176
Software Manager utilities, 250
backup-swmanager.pl, 250 System sanity, BroadWorks system monitoring, 230
restore-swmanager.pl, 250 Take server out-of-service for maintenance
Software version management, 88 Non-redundant systems, 81
Active software versions, 89 Redundant systems, 81
Display active software versions, 88 Server management, 81
List installed software versions, 88 Tech-support
Solaris DiskSuite maintenance Scheduled maintenance tasks, 98
Background, 175 Server state monitoring tools, 86
Metadatabase replica status, 176 Third-party components, uninstalling obsolete, 157
Procedures for maintenance and recovery, 176 Thread activity, platform, 190
Replacement disk requirements, 175 Threshold alarms
Submirror status, 175 Attributes, 60
Solaris DiskSuite, maintenance, 175 Configure through SNMP, 63
Solaris system logs, 55 Configure using CLI, 61
Solution, DST adjustments, 252 Fault and performance measurements, 60
Limitation for Application Server thresholds, 64
SSH keys, creating, 135
Time zone support, BroadWorks Application Server,
SSH, using, 137
235
SSH/RSYNC configuration
Configure SSH between servers, 135 TimesTen
Create SSH keys, 135 Connection optimization, additional TimesTen
Redundancy management, 135 maintenance, 110
SSH, 137 Database statistics, 107
Database synchronization, Application Server
Start
and Network Server database replication,
BroadWorks, 77
113
BroadWorks application, 77
Logs, 55
Start and stop Replication monitoring, Application Server and
Database daemon, additional TimesTen Network Server database replication, 115
maintenance tasks, 107 Replication port number, Application Server and
MySQL database daemon, additional MySQL Network Server database replication, 117
maintenance tasks, 111, 112
Tomcat logs, 55
Start server
ttStatus, scripts and utilities, 248
Start BroadWorks, 77 Undeployment, Application management, 74
Start BroadWorks application, 77 Uninstall
startbw, scripts and utilities, 246 BroadWorks application on Xsp, 76
Obsolete BroadWorks versions, 156
Statistics, TimesTen database, 107
Obsolete third-party components, obsolete, 157
Stop
UNIX command execution logging, 54
BroadWorks, 77
BroadWorks application, 78 Unlock Network Server or Application Server, 80
Server without locking, 78 Update
Stop server Application Server and Network Server address
Server management, 77 on OCS, change IP address or host name,
Stop BroadWorks, 77 170
Stop BroadWorks application, 78 Application Server call processing routing
Without locking, 78 information, 165
Application Server IP address on Network
stopbw, scripts and utilities, 247
Server
Strategy, disaster recovery, 138 Change IP address or host name, 166
Submirror status, Solaris DiskSuite, 175 BroadWorks network configuration
synchcheck_basic.pl, scripts and utilities, 248 Change IP address or host name, 164
synchcheck_full.pl, scripts and utilities, 248 HTTP Server network configuration, 164
Restart configuration agent, 164

©
EMS redundancy configuration, changing IP Additional TimesTen maintenance tasks,
address or host name, 172 107
Execution Server IP Address on Network TimesTen database statistics, additional
Server, 173 TimesTen maintenance tasks, 107
Media Server IP address on Application Server, Xtended Services Platform IP address on
changing IP address or host name, 170 Network Server, changing IP address or
Media Server IP address on Network Server, host name, 171
changing IP address or host name, 169 Versions of BroadWorks, uninstalling obsolete, 156
Network Server IP address on Application View
Server, changing IP address or host name, Current maintenance state, server maintenance
167 state, 87
Network Server or Application Server address Current position in MIB, performance
on Xtended Services Platform, changing IP measurements, 66
address or host name, 172 MIB node values, performance measurements,
Network Server routing alias list, changing IP 66
address or host name, 168 Next MIB node, performance measurements,
Notification subsystem configuration, 173 65
OCI provisioning network access list on Profile System alarms, 57
Server, 173
View and set CLI alarm backlog size, 58
Profile Server IP Address on Network Server,
173 Web.xml file, 154
TimesTen database paging information

©

BW MaintenanceGuide R200

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BW MaintenanceGuide R200

Uploaded by

Copyright:

Available Formats

Maintenance Guide

9737 Washingtonian Boulevard, Suite 350

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Release Version Reason for Change Date Author

14.0 1 Edited changes. June 9, 2006 Patricia Renaud

14.0 1 Added index. July 21, 2006 Andrea Fitzwilliam

14.0 1 Edited document. August 24, 2006 Patricia Renaud

14.0 2 Edited changes. October 17, 2006 Patricia Renaud

14.0 3 Edited changes. March 16, 2007 Patricia Renaud

14.0 4 Edited changes. May 21, 2007 Patricia Renaud

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

14.0 10 Added a procedure for importing a October 1, 2007 Tony Pilote

14.0 12 Modified section 9 Scheduled Maintenance March 6, 2008 Jean-Francois

14.0 13 Created section 10.5.7 TimesTen April 2, 2008 Roberta Boyle

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

15.0 2 Modified startFileRep.sh section to October 9, 2008 Joël Collin

15.0 3 Added section 21.14.1 importdb.pl on December 2, 2008 Goska Auerbach

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

17.0 2 Removed section on Optimization April 27, 2010 Goska Auerbach

17.0 4 Reviewed changes. September 9, 2010 Patricia Renaud

17.0 5 Edited and published document. January 6, 2011 Patricia Renaud

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

17.0 7 Updated section 11.5.2 Element May 9, 2011 Mario Goupil

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

18.0 4 Updated section 12.2.3 BroadWorks February 1, 2012 Goska Auerbach

18.0 7 Updated section 16 Change IP Address, August 6, 2012 François Isabelle

18.0 8 Reviewed changes. September 5, 2012 Patricia Renaud

18.0 8 Reviewed changes. September 14, 2012 Goska Auerbach

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

19.0 1 Update section 12 Disaster Recovery October 1, 2012 François Isabelle

19.0 3 Updated section 9 Scheduled Maintenance September 18, 2013 Jean-Francois

20.0 1 Updated sections 19.2.1.3.1 Real-Time December 4, 2013 Goska Auerbach

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

20.0 5 Edited changes. May 30, 2014 Jessica Boyle

20.0 6 Updated section 14 Install BroadWorks December 8, 2014 Hang Tran

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

20.0 12 Updated section 19.4.2 Redundancy April 4, 2017 Goska Auerbach

20.0 12 Updated section 16 Change IP Address, July 7, 2017 Goska Auerbach

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

2.1 Changes for Release 20.0

Release 20.0, Version 13

Release 20.0, Version 12

Release 20.0, Version 11

Release 20.0, Version 10

Release 20.0, Version 9

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Release 20.0, Version 8

Release 20.0, Version 7

Release 20.0, Version 6

Release 20.0, Version 5

Release 20.0, Version 4

Release 20.0, Version 3

Release 20.0, Version 2

Release 20.0, Version 1