Fast Application Notification with Oracle Restart

(from Oracle Database Administrators Guide Chapter 4)

In a standalone server environment, Oracle Restart uses Oracle Notification Services
(ONS) and Oracle Advanced Queues to publish Fast Application Notification (FAN)
high availability events. Integrated Oracle clients use FAN to provide fast notification
to clients when the service or instance goes down. The client can automate the failover
of database connections between a primary database and a standby database.
This section describes how ONS and FAN work with Oracle Restart. It contains the
following topics:
Overview of Fast Application Notification
Application High Availability with Services and FAN
Managing Unplanned Outages
Managing Planned Outages
Fast Application Notification High Availability Events
Using Fast Application Notification Callouts
Oracle Clients That Are Integrated with Fast Application Notification

Overview of Fast Application Notification
FAN is a notification mechanism that Oracle Restart can use to notify other processes
about configuration changes that include service status changes, such as UP or DOWN
events. FAN provides the ability to immediately terminate inflight transaction when
an instance or server fails. Integrated Oracle clients receive the events and respond.
Applications can respond either by propagating the error to the user or by
resubmitting the transactions and masking the error from the application user. When a
DOWN event occurs, integrated clients immediately clean up connections to the
terminated database. When an UP event occurs, the clients create new connections to
the new primary database instance.
Oracle Restart publishes FAN events whenever a managed instance or service goes up
or down. After a failover, the Oracle Data Guard Broker (broker) publishes FAN
events. These FAN events can be used in the following ways:

Applications can use FAN with Oracle Restart without programmatic changes if
they use one of these Oracle integrated database clients: Oracle Database JDBC,
Universal Connection Pool for Java, Oracle Call Interface, and Oracle Database
ODP.NET. These clients can be configured for Fast Connection Failover (FCF) to
automatically connect to a new primary database after a failover.

FAN server-side callouts can be configured on the database tier.
For DOWN events, such as a failed primary database, FAN provides immediate
notification to the clients so that they can failover as fast as possible to the new
primary database. The clients do not wait for a timeout. The clients are notified
immediately, and they must be configured to failover when they are notified.

For UP events, when services and instances are started, new connections can be created
so that the application can immediately take advantage of the extra resources.

Through server-side callouts, you can also use FAN to:
Log status information
Page DBAs or open support tickets when resources fail to start
Automatically start dependent external applications that must be co-located with a service

FAN events are published using ONS and Oracle Streams Advanced Queuing queues.
The queues are configured automatically when you configure a service. You must
configure ONS manually using SRVCTL commands.

The Connection Manager (CMAN) and Oracle Net Services listeners are integrated
with FAN events, enabling the CMAN and the listener to immediately de-register
services provided by the failed instance and to avoid erroneously sending connection
requests to a failed database.

Application High Availability with Services and FAN
Oracle Database focuses on maintaining service availability. With Oracle Restart,
Oracle services are designed to be continuously available. Oracle Restart monitors the
database and its services and, when configured, sends event notifications using FAN.

Managing Unplanned Outages If Oracle Restart detects an outage, then it isolates the
failed component and recovers the dependent components. If the failed component is
the database instance, then after Oracle Data Guard fails over to the standby database,
Oracle Restart on the new primary database starts any services defined with the
current role.

FAN events are published by Oracle Restart and the Oracle Data Guard Broker
through ONS and Advanced Queuing. You can also perform notifications using FAN
callouts.

With Oracle Restart, restart and recovery are automatic, including the restarting of the
subsystems, such as the listener and the Oracle Automatic Storage Management
(Oracle ASM) processes, not just the database. You can use FAN callouts to report
faults to your fault management system and to initiate repair jobs.

Managing Planned Outages For repairs, upgrades, and changes that require you to shut
down the primary database, Oracle Restart provides interfaces that disable and enable
services to minimize service disruption to application users. Using Oracle Data Guard
Broker with Oracle Restart allows a coordinated failover of the database service from
the primary to the standby for the duration of the planned outage. Once you complete
the operation, you can return the service to normal operation.

The management policy for a service controls whether the service starts automatically
when the database is restarted. If the management policy for a service is set to
AUTOMATIC, then it restarts automatically. If the management policy for a service is set
to MANUAL, then it must be started manually.

Fast Application Notification High Availability Events Table 4–4 describes the FAN event
record parameters and the event types, followed by name-value pairs for the event
properties. The event type is always the first entry and the timestamp is always the
last entry. In the following example, the name in the name-value pair is shown in Fan
event type (service_member), and the value in the name-value pair is shown in
Properties:

FAN event type: service_member
Properties: version=1.0 service=ERP database=FINPROD instance=FINPROD host=node1
status=up

Table 4-4
VERSION    Version of the event record. Used to identify release changes.
EVENT TYPE   SERVICE, SERVICE_MEMBER, DATABASE, INSTANCE, NODE, ASM, SRV_PRECONNECT.
Note that database and Instance types provide the database service, such as DB_UNIQUE_NAME.DB_DOMAIN.
DATABASE UNIQUE NAME  The unique database supporting the service; matches the
initialization parameter value for DB_UNIQUE_NAME, which
defaults to the value of the initialization parameter DB_NAME.
INSTANCE   The name of the instance that supports the service; matches the ORACLE_SID value.
NODE NAME   The name of the node that supports the service or the node that has stopped; matches the node name known to Cluster
Synchronization Services (CSS).
SERVICE   The service name; matches the service in DBA_SERVICES.
STATUS    Values are UP, DOWN, NOT_RESTARTING, PRECONN_UP, PRECONN_DOWN, and UNKNOWN.
REASON    Data_Guard_Failover, Failure, Dependency, User, Autostart, Restart.
CARDINALITY   The number of service members that are currently active; included in all UP events.
TIMESTAMP   The local time zone to use when ordering notification events.

A FAN record matches the database signature of each session as shown in Table 4-5
Table 4-5
SERVICE   sys_context(‘userenv’, ‘service_name’)
DATABASE UNIQUE NAME  sys_context(‘userenv’, ‘db_unique_name’)
INSTANCE   sys_context(‘userenv’, ‘instance_name’)
NODE NAME   sys_context(‘userenv’, ‘server_host’)

Using Fast Application Notification Callouts FAN callouts are server-side executables that
Oracle Restart executes immediately when high availability events occur. You can use
FAN callouts to automate the following activities when events occur, such as:
Opening fault tracking tickets
Sending messages to pagers
Sending e-mail
Starting and stopping server-side applications
Maintaining an uptime log by logging each event as it occurs

To use FAN callouts, place an executable in the directory grid_home/racg/usrco on
both the primary and the standby database servers. If you are using scripts, then set
the shell as the first line of the executable. The following is an example file for the
grid_home/racg/usrco/callout.sh callout:

#! /bin/ksh
FAN_LOGFILE= [your path name]/admin/log/`hostname`_uptime.log
echo $* “reported=”`date` >> $FAN_LOGFILE &

The following output is from the previous example:
NODE VERSION=1.0 host=sun880-2 status=nodedown reason=
timestamp=08-Oct-2004 04:02:14 reported=Fri Oct 8 04:02:14 PDT 2004

A FAN record matches the database signature of each session, as shown in Table 4-5
Use this information to take actions on sessions that match the FAN event data.

Oracle Clients That Are Integrated with Fast Application Notification Oracle has integrated
FAN with many of the common Oracle client drivers that are used to connect to Oracle
Restart databases. Therefore, the easiest way to use FAN is to use an integrated Oracle
Client.

You can use the CMAN session pools, Oracle Call Interface, Universal Connection
Pool for Java, JDBC simplefan API, and ODP.NET connection pools. The overall goal is
to enable applications to consistently obtain connections to the available primary
database at anytime.