SYS_LOB {10 digit object_id} C {5 digit col#} $$

We can extract the object_id 164261 from the segment_name SYS_LOB0000164261C00005. Query on dba_objects shows the lob segment locates in the table SYS.SQLOBJ$DATA.

testsupp@testPRD> @df SYSAUX

TABLESPACE_NAME         TotalMB     UsedMB     FreeMB % Used
-------------------- ---------- ---------- ---------- ------
SYSAUX                     4072       4072          0   100%
                     ---------- ----------
                           4072       4072

testsupp@testPRD> @topseg sysaux

TABLESPACE_NAME      OWNER   SEGMENT_NAME               SEGMENT_TYPE      MB
-------------------- ------- -------------------------- ------------- ------
SYSAUX               SYS     SYS_LOB0000164261C00005$$  LOBSEGMENT      1855
SYSAUX               SYS     WRH$_SQL_PLAN              TABLE            363
SYSAUX               SYS     SYS_LOB0000008981C00004$$  LOBSEGMENT       341
....

50 rows selected.

testsupp@testPRD> @o2 164261

OWNER OBJECT_TYPE  OBJECT_ID DATA_OBJECT_ID OBJECT_NAME  STATUS LAST_DDL_TIME
----- ----------- ---------- -------------- ------------ ------ -------------------
SYS   TABLE           164261                SQLOBJ$DATA  VALID  2011-08-07 19:51:30

testsupp@testPRD> @lob sys.SQLOBJ$DATA

TABLE_NAME   COLUMN_NAME SEGMENT_NAME              TABLESP INDEX_NAME
------------ ----------- ------------------------- ------- ------------------------
SQLOBJ$DATA  SPARE2      SYS_LOB0000164261C00007$$ SYSAUX  SYS_IL0000164261C00007$$
SQLOBJ$DATA  COMP_DATA   SYS_LOB0000164261C00005$$ SYSAUX  SYS_IL0000164261C00005$$

sys@CS11GR2> @ddl sys.SQLOBJ$DATA

CREATE TABLE SYS.SQLOBJ$DATA
(    SIGNATURE NUMBER,
     CATEGORY VARCHAR2(30),
     OBJ_TYPE NUMBER,
     PLAN_ID NUMBER,
     COMP_DATA CLOB NOT NULL ENABLE,
     SPARE1 NUMBER,
     SPARE2 CLOB,
     CONSTRAINT SQLOBJ$DATA_PKEY PRIMARY KEY (SIGNATURE, CATEGORY, OBJ_TYPE, PLAN_ID) ENABLE
) ORGANIZATION INDEX ......

1 row selected.

Search SQLOBJ$DATA on MOS I find the bug 9910484, the size of the table SQLOBJ$data is probably related to the baseline capture. Query on v$sql releases there are excess executions on the sql 1vxm21mhmgy07, which merges the baseline data into SQLOBJ$DATA. I enable 10046 trace and run some query, the sql 1vxm21mhmgy07 is issued during every hard parse. This explains the excess executions on 1vxm21mhmgy07 and the size of the lob segment SYS_LOB0000164261C00005$$. The production DB does not have such issue because the optimizer_capture_sql_plan_baselines is set to false.

BUG:9910484 – UNNECESSARY UPDATES ON SQLOBJ$DATA CAUSING OBJECT AND TABLESPACE (SYSAUX) GROWTH

testsupp@testPRD> show parameter baseline

NAME                                 TYPE     VALUE
------------------------------------ -------- ------
optimizer_capture_sql_plan_baselines boolean  TRUE
optimizer_use_sql_plan_baselines     boolean  TRUE

testsupp@testPRD> @sqlt SQLOBJ$DATA

SQL_ID        CH#       PLAN       EXEC  rows/exec ela_tm(cs)/exec  gets/exec reads/exec
------------- --- ---------- ---------- ---------- --------------- ---------- ----------
7xa8wfych4mad   0 2615480013          1          1              13          0          0
1vxm21mhmgy07   0 3193071292         24          1               5         28          0
1vxm21mhmgy07   1 3193071292     146622          1               0         39          0
...

29 rows selected.

==============================================================================
The many versions of 1vxm21mhmgy07 is due to multi users connect to DB.
==============================================================================
testsupp@testPRD> @sql 1vxm21mhmgy07

SQL_ID        CH#       PLAN       EXEC  rows/exec ela_tm(cs)/exec  gets/exec reads/ex
------------- --- ---------- ---------- ---------- --------------- ---------- ----------
1vxm21mhmgy07   1 3193071292     146622          1               0         39          0
1vxm21mhmgy07   4 3193071292     130269          0               1        190          0
1vxm21mhmgy07  11 3960238002         19          1               1         48          1
1vxm21mhmgy07   9 3960238002       9082          1               1         67          1
1vxm21mhmgy07   2 3193071292         23          1               2         42          0
1vxm21mhmgy07   6 3960238002      11485          0               2         92          1
1vxm21mhmgy07  10 3960238002         23          1               2         49          1
1vxm21mhmgy07   8 3960238002       4435          0               2         79          1
1vxm21mhmgy07   7 3960238002          4          1               2         55          1
1vxm21mhmgy07   0 3193071292         24          1               5         28          0
1vxm21mhmgy07  12 3960238002          8          1               9        116          3
1vxm21mhmgy07   3 3193071292         95          1              17         23          0
1vxm21mhmgy07   5 3960238002          2          1              46        109          2

13 rows selected.

testsupp@testPRD> @sqlf 1vxm21mhmgy07

SQL_FULLTEXT
----------------------------------------------------------------------------------------------------
MERGE INTO sqlobj$data USING dual ON (:1 IS NULL) WHEN MATCHED THEN UPDATE SET
  comp_data = :2
WHERE signature = :3 AND category = :4 AND obj_type = :5 AND plan_id = :6 WHEN
  NOT MATCHED THEN INSERT (signature, category, obj_type, plan_id, comp_data,
  spare1, spare2) VALUES (:7, :8, :9, :10, :11, null, null)

==========================
From the 10046 trace file
==========================

PARSING IN CURSOR #3063571832 len=901 dep=1 uid=0 oct=189 lid=0 tim=1329804200425559 hv=3778541575 ad='2f34fdc8' sqlid='1vxm21mhmgy07'
MERGE INTO sqlobj$data ...

Solution
1. Disable the baseline capture.

alter system set optimizer_capture_sql_plan_baselines=false scope=both;

2. Extend the sysaux to 5G.

Why does Oracle need to take care the tablescan seriously?

If you scan a large object, you could flush a huge amount of useful data from the cache,data that you may then have to reread very promptly. Tablescans shouldnt really happen often in OLTP systems, and when they do happen, you need to ensure that they dont cause problems. The potential for performance problems relating to tablescans resulted in Oracle Corp. writing code to distinguish between “short and “long tables (and, recently, “medium tables, although there are no statistics collected to record that particular option).

For the short tables [1, 2]:

There is still a 2 percent limit (dictated by the hidden parameter _small_table_threshold) and the blocks are still loaded into the data cache in the normal way, but the touch count is incremented for these buffers.

For the medium tables [2, 10]:

The second case has a 10 percent limit (though I dont know how its set) where the tablescan is initially considered to be a long tablescan, so touch counts will not be incremented as the blocks are read into buffers and (most of) the buffers will immediately be moved to REPL_AUX; however, if you repeat the tablescan while the blocks are still in the data cache, the touch count on the buffers will be incremented at that point and the tablescan will be reported as a short tablescan.

For the long table: [10, ∞]

The final case has a limit of 25 percent where the touch count is never incremented and the buffers are cycled to REPL_AUX very quickly—basically you do a multiblock read pulling a few buffers from REPL_AUX, you do the next multiblock read pulling more buffers from REPL_AUX but pushing the first batch of tablescanned buffers into REPL_AUX, and keep repeating this cycle. For a very large table you will end up with the entire REPL_AUX loaded with blocks from that table, and a small number of blocks from the table in REPL_MAIN. Thus, Oracle protects a very large fraction of your data cache from overaggressive tablescans.

One month ago I tried to setup a test case in 10.2.0.4 to verify what Jonathan suggested. The result does not match the description in the book. After feedback to Jonathan, the difference is found because I did not actually fill up the buffer pool in advance. In such a case, the threshhold for short tables is 10 percent, rathan than 2 percent. To simulate normal behavior, the buffer pool need to be filled up first. To fill up the buffer pool, We cannot simply do a big tablescan. A convenient way is a index scan on a big table. Jonathan already designed a test case, the blog post of small tables. I decided to repeat the test case on 11.2.0.2.0 Linux 32 bit. I think it’s worth a blog post.

Here is steps to verify:
1. Create the tables as 2, 5, 10, 25, 50, 100 percent of the buffer pool, gather the statistics on the table.
2. Flush the buffer pool;
3. Fill up the buffer pool by index scan on the table t[100], by “db file sequential read”.
4. Scan the table t[2] 3 times.
5. Scan the table t[5] 3 times.
6. Scan the table t[10] 3 times.
7. Scan the table t[25] 3 times.
8. Scan the table t[50] 3 times.

After every tablescan, sleep 4 seconds to allow the TCH increment, and check the number of buffers to cache the table and it’s TCH. We will monitor below session statistics.
physical reads
physical reads cache
physical reads direct
table scans (long tables)
table scans (direct read)
table scans (short tables)

Here is the step by step log.

1. Create the tables as 2, 5, 10, 25, 50, 100 percent of the buffer pool, gather the statisitics on the table. The _small_table_threshold is default 2 percent of buffer_pool.

select
	buffers,
	buffers/50 "2_percent",
	buffers/20 "5_percent",
	buffers/10 "10_percent",
	buffers/4 "25_percent",
	buffers/2 "50_percent"
from
	v$buffer_pool;

   BUFFERS  2_percent  5_percent 10_percent 25_percent 50_percent
---------- ---------- ---------- ---------- ---------- ----------
     14880      297.6        744       1488       3720       7440

sys@CS11GR2> @pd small_table

NAME                    VALUE DESCRIPTION
----------------------- ----- -------------------------------------------------------
_small_table_threshold  297   lower threshold level of table size for direct reads

============================
Create 6 tables
============================
create table t_297
pctfree 99
pctused 1
as
with generator as (
	select	--+ materialize
		rownum id
	from dual
	connect by
		rownum <= 10000
)
select
	rownum			id,
	lpad(rownum,10,'0')	small_vc,
	rpad('x',100)		padding
from
	generator	v1,
	generator	v2
where
	rownum <= 297
;

create table t_744  ...
create table t_1488  ...
create table t_3720  ...
create table t_7440  ...
create table t_14880  ... 

create index t_14880_id on t_14880(id);

===================================
Gather the statistics for 6 tables
===================================

begin
	dbms_stats.gather_table_stats(
		ownname		 => user,
		tabname		 =>'T_297',
		estimate_percent => 100,
		method_opt 	 => 'for all columns size 1'
	);
end;
/

.....

===================================
Query the data_object_id
===================================
select
	object_name, object_id, data_object_id
from
	user_objects
where
	object_name in  (
		'T_297',
		'T_744',
		'T_1488',
		'T_3720',
		'T_7440',
		'T_14880',
		'T_14880_ID'
	)
order by
	object_id
;

OBJECT_NAME                     OBJECT_ID DATA_OBJECT_ID
------------------------------ ---------- --------------
T_297                               87014          87014
T_744                               87015          87015
T_1488                              87016          87016
T_3720                              87017          87017
T_7440                              87018          87018
T_14880                             87019          87019
T_14880_ID                          87020          87020

7 rows selected.

2. Flush the buffer pool;

sid@CS11GR2> alter system flush buffer_cache;

System altered.

3. Fill up the buffer pool by index scan on the table t_14880. Although the query run 3 times, the TCH is not incremented.

sid@CS11GR2> set autotrace on
sid@CS11GR2> select
  2  	     /*+ index(t) */
  3  	     max(small_vc)
  4  from
  5  	     t_14880 t
  6  where
  7  	     id > 0
  8  ;

MAX(SMALL_VC)
----------------------------------------
0000014880

Execution Plan
----------------------------------------------------------
Plan hash value: 2029532131

-------------------------------------------------------------------------------------------
| Id  | Operation                    | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |            |     1 |    16 | 14919   (1)| 00:03:00 |
|   1 |  SORT AGGREGATE              |            |     1 |    16 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| T_14880    | 14880 |   232K| 14919   (1)| 00:03:00 |
|*  3 |    INDEX RANGE SCAN          | T_14880_ID | 14880 |       |    33   (0)| 00:00:01 |
-------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - access("ID">0)

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
      14913  consistent gets
      14915  physical reads
          0  redo size
        435  bytes sent via SQL*Net to client
        420  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

sid@CS11GR2> select
  2  	     obj, tch, count(*)
  3  from    x$_bh
  4  where
  5  	     obj between 87014 and 87020
  6  group by
  7  	     obj, tch
  8  order by
  9  	     count(*)
 10  ;

       OBJ        TCH   COUNT(*)
---------- ---------- ----------
     87020          1         32
     87019          1      14844

The table scan is wrapped with snap_my_stats to capture the snapshot of the session statistics. Query the content of the buffer pool after every 3 scans.

exec snap_my_stats.start_snap;
select
	max(small_vc)
from
	t_297 t;
exec snap_my_stats.end_snap;

exec dbms_lock.sleep(4);

exec snap_my_stats.start_snap;
select
	max(small_vc)
from
	t_297 t;
exec snap_my_stats.end_snap;

exec dbms_lock.sleep(4);

exec snap_my_stats.start_snap;
select
	max(small_vc)
from
	t_297 t;
exec snap_my_stats.end_snap;

select
	obj, tch, count(*)
from	x$_bh
where
	obj between 87014 and 87020
group by
	obj, tch
order by
	count(*);

4. Scan the table T_297 3 times(2 percent). All the blocks are cached and the TCH is incremented on every access.

Name                                                                     Value
----                                                                     -----
physical reads                                                             311
physical reads cache                                                       311
table scans (short tables)                                                   1

Name                                                                     Value
----                                                                     -----
table scans (short tables)                                                   1

Name                                                                     Value
----                                                                     -----
table scans (short tables)                                                   1

       OBJ        TCH   COUNT(*)
---------- ---------- ----------
     87020          1         32
     87014          3        298	<-- the blocks of T_297
     87019          1      14479

5. Scan the table T_744 3 times(5 percent). The first scan is by direct path as long tables. The follow two scans are handled as short tables. After 3 scans, the TCH of the segment header is 3, the TCH of the data blocks is 2.

Name                                                                     Value
----                                                                     -----
physical reads                                                             745
physical reads cache                                                         1
physical reads direct                                                      744
table scans (long tables)                                                    1
table scans (direct read)                                                    1

Name                                                                     Value
----                                                                     -----
physical reads                                                             744
physical reads cache                                                       744
table scans (short tables)                                                   1

Name                                                                     Value
----                                                                     -----
table scans (short tables)                                                   1

       OBJ        TCH   COUNT(*)
---------- ---------- ----------
     87015          3          1	<-- the segment header of T_744
     87020          1         30
     87014          3        298
     87015          2        744	<-- the data blocks of T_744
     87019          1      13736

6. Scan the table T_1488 3 times(10 percent). The 3 scans are by direct path as long tables, except the segment header block, no blocks is cached.

Name                                                                     Value
----                                                                     -----
physical reads                                                           1,490
physical reads cache                                                         2
physical reads direct                                                    1,488
table scans (long tables)                                                    1
table scans (direct read)                                                    1

Name                                                                     Value
----                                                                     -----
physical reads                                                           1,488
physical reads direct                                                    1,488
table scans (long tables)                                                    1
table scans (direct read)                                                    1

Name                                                                     Value
----                                                                     -----
physical reads                                                           1,488
physical reads direct                                                    1,488
table scans (long tables)                                                    1
table scans (direct read)                                                    1

       OBJ        TCH   COUNT(*)
---------- ---------- ----------
     87015          3          1
     87016          3          1	<-- the segment header of T_1488
     87020          1         30
     87014          3        298
     87015          2        744
     87019          1      13734

7. Start from 10 percent, the behavior is the same. The statistics for tables T_3720 and T_7440 is the same pattern as T_1488.

So, the result matches the description in the book. The optimizer makes the decision base on the table statistics, that’s the number of blocks in this case. If the numlblks of T_297 is updated to 744 and 1488, manually by dbms_stat.set_table_stats. The behavior of tablescan on T_297 is the same as T_744 and T_1488. This is another reason why DBA need to keep on eye closely an the statistics.

SQL> select     name, level#
  2  from       v$latch
  3  where      name in ('cache buffers lru chain','cache buffers chains')
  4  /

NAME                               LEVEL#
-------------------------------    ------
cache buffers lru chain                 2
cache buffers chains                    1

2 rows selected.

The cache buffers chains latch has a lower level than the cache buffers lru chain latch, so we can’t request the cache buffers lru chain latch in willing-to-wait mode if we’re already holding the cache buffers chains latch. Think about what this means: we’re holding the cache buffers chains latch (which I will call the hash latch for the rest of this subsection) because we’ve just searched the hash chain for a buffer and discovered that, for whatever reason, we need to add another buffer to the chain. So we have to acquire the cache buffers lru chain latch (which I will call the lru latch for the rest of this subsection) to move a buffer from the REPL_AUX list to the midpoint of the REPL_MAIN list; but we can’t request it in willing-to-wait mode because we’re already holding a lower-level latch.

…
But if you can’t get the lru latch with an immediate get, you have to drop the hash latch, get the lru latch, and then get the hash latch again.
…

(I’ll refer the “cache buffers lru chain” as lru latch, and “cache buffers chains” as hash latch in this blog)
To avoid latch deadlock, Oracle acquire the latches by the order of ascending level#. After holding the hash latch(level#=1), Oracle will acquire the lru latch(level#=2) in willing-to-wait mode. The similar procedure can be observed on “library cache”(level=5) and “shared pool”(level#=7) latch in 10g. Cuihua use below method to verify that in fact, Oracle will acquire the lru latch after holding the hash latch, opposite to what Jonathan suggests.
Open 3 sessions:
1. Session 1 hold all the cache buffers lru chain latch;
2. After session 1 hold all the lru latch, session 2 issue an update and hang, the update need to do eithor physical reads or switch current to new buffer, to make sure the session 2 need to acquire the lru latch.
3. Session 3 dump the process state of session 2, from the trace file we can find the latch holding/waiting information.

In the dump file from Cuihua, session 2 is found waiting on the lru latch, the the same time, holding the hash latch. Oracle did not drop the hash latch before requing lru latch.

waiting for 3493da3c Child cache buffers lru chain level=2 child#=11
holding    (efd=23) 34848c10 Child cache buffers chains level=1 child#=2313

There are two method to manually hold a latch:
1. oradebug poke 1, to set the value of latch address to 1
2. oradebug call kslgetl 1, to simulate the latch get call.

Cuihua use the oradebug poke method in his blog. The kslgetl call method will increase the latch statistics, just as normal latch activity. While oradebug poke is more like backdoor hacking, it changes the value in memory directly, holds the latch silently, and does not increase the latch gets. The difference prompt me to repeat Cuihua‘s test case, using the kslgetl call method, to check if there is any variation.

Here is a small test on the two idle lru latches to show the difference. 0x39982AF8 and 0x39982F98 are picked up for testing. Only the gets of latch 0x39982AF8 moved from 0 to 1. The testing is done on 11.2.0.2.0 on Linux 32 bit.

sys@CS11GR2> select * from (select addr,to_number(addr,'XXXXXXXXXXXX') addr_dec,gets,misses,immediate_gets,immediate_misses
	from v$latch_children where name = 'cache buffers lru chain' order by addr asc) where rownum<5;

ADDR       ADDR_DEC       GETS     MISSES IMMEDIATE_GETS IMMEDIATE_MISSES
-------- ---------- ---------- ---------- -------------- ----------------
39982A74  966273652         36          0              0                0
39982AF8  966273784          0          0              0                0
39982F14  966274836         36          0              0                0
39982F98  966274968          0          0              0                0

sys@CS11GR2> oradebug setmypid
Statement processed.
sys@CS11GR2> oradebug peek 0x39982AF8 300
[39982AF8, 39982C24) = 00000000 00000000 00000096 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug call kslgetl 966273784 1
Function returned 1
sys@CS11GR2> oradebug peek 0x39982AF8 300
[39982AF8, 39982C24) = 00000011 00000001 00000096 00000002 00000001 BFBA01EF 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug call kslfre 966273784 1
Function returned 0
sys@CS11GR2> oradebug peek 0x39982AF8 300
[39982AF8, 39982C24) = 00000000 00000001 00000096 00000002 00000001 BFBA01EF 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2>
sys@CS11GR2> oradebug peek 0x39982F98 300
[39982F98, 399830C4) = 00000000 00000000 00000096 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug poke 0x39982F98 4 0x00000001
BEFORE: [39982F98, 39982F9C) = 00000000
AFTER:  [39982F98, 39982F9C) = 00000001
sys@CS11GR2> oradebug peek 0x39982F98 300
[39982F98, 399830C4) = 00000001 00000000 00000096 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug poke 0x39982F98 4 0x00000000
BEFORE: [39982F98, 39982F9C) = 00000001
AFTER:  [39982F98, 39982F9C) = 00000000
sys@CS11GR2> oradebug peek 0x39982F98 300
[39982F98, 399830C4) = 00000000 00000000 00000096 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2>
sys@CS11GR2> select * from (select addr,to_number(addr,'XXXXXXXXXXXX') addr_dec,gets,misses,immediate_gets,immediate_misses
		from v$latch_children where name = 'cache buffers lru chain' order by addr asc) where rownum<5;

ADDR       ADDR_DEC       GETS     MISSES IMMEDIATE_GETS IMMEDIATE_MISSES
-------- ---------- ---------- ---------- -------------- ----------------
39982A74  966273652         36          0              0                0
39982AF8  966273784          1          0              0                0
39982F14  966274836         36          0              0                0
39982F98  966274968          0          0              0                0

The procedure is basically the same, to confirm that Oracle will acquire the lru latch while holding the hash latch.
Preparation.
1. Create a table for switch current to new buffer. When update by tablescan, the session will move a buffer from auxiliary replacement list to the main list, thus need to acquire the lru latch first.
2. Query the active lru latches
3. Get the spid of session 2
4. Preparation in session 3 do a process dump

Verification.
1. In session 1, hold the active lru latches.
2. In session 2, issue update on the table T
3. In session 3. dump the process state of session 2

Step by step output.
Preparation:
1. Create a table for switch current to new buffer.

sys@CS11GR2> drop table t purge;

Table dropped.

sys@CS11GR2> create table t (id number, padding varchar2(1000));

Table created.

sys@CS11GR2> insert into t values(1,lpad('1',1000,'0'));

1 row created.

sys@CS11GR2> commit;

Commit complete.

2. Query the active lru latches from x$kcbwd, we can see the two active lru latch are 0x399dc524 and 0x399dc9c4.

sys@CS11GR2> select
  2  	     SET_ID,
  3  	     DBWR_NUM,
  4  	     SET_LATCH,
  5  	     CNUM_REPL,
  6  	     ANUM_REPL,
  7  	     'oradebug call kslgetl ' || to_number(SET_LATCH,'xxxxxxxxxxxx') || ' 1' get_statment
  8  from
  9  	     x$kcbwds;

    SET_ID   DBWR_NUM SET_LATC  CNUM_REPL  ANUM_REPL GET_STATMENT
---------- ---------- -------- ---------- ---------- ----------------------------------------------------------------
         1          0 39982A74          0          0 oradebug call kslgetl 966273652 1
         2          0 39982F14          0          0 oradebug call kslgetl 966274836 1
         3          0 399AF7CC          0          0 oradebug call kslgetl 966457292 1
         4          0 399AFC6C          0          0 oradebug call kslgetl 966458476 1
         5          0 399DC524       7440       3322 oradebug call kslgetl 966640932 1
         6          0 399DC9C4       7440       3341 oradebug call kslgetl 966642116 1
         7          0 39A0927C          0          0 oradebug call kslgetl 966824572 1
         8          0 39A0971C          0          0 oradebug call kslgetl 966825756 1
         9          0 39A35FD4          0          0 oradebug call kslgetl 967008212 1
        10          0 39A36474          0          0 oradebug call kslgetl 967009396 1
        11          0 39A62D2C          0          0 oradebug call kslgetl 967191852 1
        12          0 39A631CC          0          0 oradebug call kslgetl 967193036 1
        13          0 39A8FA84          0          0 oradebug call kslgetl 967375492 1
        14          0 39A8FF24          0          0 oradebug call kslgetl 967376676 1
        15          0 39ABC7DC          0          0 oradebug call kslgetl 967559132 1
        16          0 39ABCC7C          0          0 oradebug call kslgetl 967560316 1

16 rows selected.

3. Get the spid of session 2.

sys@CS11GR2> select spid from v$process where addr in (select paddr from v$session where sid in (select sid from v$mystat where rownum=1));

SPID
------------------------
6741

4. preparation in session 3

sys@CS11GR2> oradebug setospid 6741
Oracle pid: 23, Unix process pid: 6741, image: oracle@cargosmart.org (TNS V1-V3)

Verification:
1. In session 1 hold the two active lru latches. ORA-600 error will be triggerred when the required lru latch is currently held. The latches holding by the session is released. The two lru latches need to be acquired from scratch, until both the first 4 bytes is changed to 0x00000011.

sys@CS11GR2> oradebug call kslgetl 966640932 1
Function returned 1
sys@CS11GR2> oradebug call kslgetl 966642116 1
ORA-00600: internal error code, arguments: [526], [0x399DC9C4], [2], [cache buffers lru chain], [11], [0x399DC524], [9], [], [], [], [], []
sys@CS11GR2> oradebug call kslgetl 966642116 1
Function returned 1
sys@CS11GR2> oradebug peek 0x399DC524 300
[399DC524, 399DC650) = 00000000 00000501 00000096 00000002 000005DC 00000000 00001725 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug peek 0x399DC9C4 300
[399DC9C4, 399DCAF0) = 00000011 00000593 00000096 00000002 00000001 BFBA01EF 0000175C 00000001 00000001 00000000 00000000 00000000 00000000 00000001 ...
sys@CS11GR2> oradebug call kslgetl 966640932 1
Function returned 1
sys@CS11GR2> oradebug peek 0x399DC524 300
[399DC524, 399DC650) = 00000011 00000502 00000096 00000002 00000001 BFBA01EF 00001725 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ...
sys@CS11GR2> oradebug peek 0x399DC9C4 300
[399DC9C4, 399DCAF0) = 00000011 00000593 00000096 00000002 00000001 BFBA01EF 0000175C 00000001 00000001 00000000 00000000 00000000 00000000 00000001 ...

2. In session 2, issue an update on the table t and the session hang.

sys@CS11GR2> update t set id = 2, padding = lpad('2',1000,'0');

3. In session 3. dump the process state of session 2.

sys@CS11GR2> oradebug dump processstate 10
Statement processed.

The process dump of session 2

PROCESS STATE
-------------
Process global information:
     process: 0x3abdbf58, call: 0x3a76ac20, xact: 0x39ba4f0c, curses: 0x3a700fc0, usrses: 0x3a700fc0
     in_exception_handler: no
  ----------------------------------------
  SO: 0x3abdbf58, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0x3abdbf58, name=process, file=ksu.h LINE:12451, pg=0
  (process) Oracle pid:23, ser:3, calls cur/top: 0x3a76ac20/0x3a76ac20
            flags : (0x0) -
            flags2: (0x0),  flags3: (0x0)
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
  (post info) last post received: 0 0 0
              last post received-location: No post
              last process to post me: none
              last post sent: 0 0 0
              last post sent-location: No post
              last process posted by me: none
    (latch info) wait_event=0 bits=2
        Location from where call was made: kcb2.h LINE:3795 ID:kcbzgb:
      waiting for 399dc524 Child cache buffers lru chain level=2 child#=9
        Location from where latch is held: kywm2.h LINE:185 ID:kywmcrpln: creating new WLM plan:
        Context saved from call: 3216638447
        state=busy [holder orapid=17] wlstate=free [value=0]
          waiters [orapid (seconds since: put on list, posted, alive check)]:
           23 (6, 1329203954, 6)
           waiter count=1
          gotten 1282 times wait, failed first 0 sleeps 0
          gotten 5925 times nowait, failed: 1
        possible holder pid = 17 ospid=6390
      on wait list for 399dc524
      holding    (efd=8) 39923cec Child cache buffers chains level=1 child#=586
        Location from where latch is held: kcb2.h LINE:3166 ID:kcbgcur_2:
        Context saved from call: 4255689
        state=busy(exclusive) [value=0x20000017, holder orapid=23] wlstate=free [value=0]
    Process Group: DEFAULT, pseudo proc: 0x3a47f3dc
    O/S info: user: oracle, term: UNKNOWN, ospid: 6741
    OSD pid info: Unix process pid: 6741, image: oracle@smart.org (TNS V1-V3)

Session 2 is waiting for “cache buffers lru chain” while holding the “cache buffers chains”.

      waiting for 399dc524 Child cache buffers lru chain level=2 child#=9
      holding    (efd=8) 39923cec Child cache buffers chains level=1 child#=586

The testing confirms Oracle will request the lru latch in willing-to-wait mode while holding the hash latch. Because of the existence of multiple active lru latches, it is possible that Oracle will get the lru latch in immediate mode; if it fails on first get, then get anothers lru latch in willing-to-wait mode. To verify if Oracle get the lru latch in immediate mode. I wrap the update in session 2 with package snap_latch, to take the snapshot of the latch statistics. After session 2 hang on the update, I go back to session 1 to release the two lru latches. Then the update on session 2 complete as expected. There is no immediate gets or misses on the lru latch, so Oracle does not try immediate mode in this case. (The snap_latch package is provided in the book)

sys@CS11GR2> set serveroutput on size 1000000 format wrapped
sys@CS11GR2> set linesize 168
sys@CS11GR2> set trimspool on
sys@CS11GR2> execute snap_latch.start_snap;

PL/SQL procedure successfully completed.

sys@CS11GR2> update t set id = 2, padding = lpad('2',1000,'0');

1 row updated.

sys@CS11GR2> execute snap_latch.end_snap;
---------------------------------
Latch waits:-   14-Feb 17:01:05
Interval:-      7 seconds
---------------------------------
Latch                              Gets      Misses     Sp_Get     Sleeps     Im_Gets   Im_Miss Holding Woken Time ms
-----                              ----      ------     ------     ------     -------   ------- ------- ----- -------
cache buffers lru chain               3           1          0          1           0         0       0     0 7,281.9

PL/SQL procedure successfully completed.

P.S. After feedback to Jonathan, He has added this into the Errata. I copy his corrections here.

Section “Loading a Hash Chain” says: “As I commented in Chapter 4, a process is not allowed to request a latch in willing-to-wait mode if it is already holding a lower-level latch.” This statement is the wrong way round – you cannot request a latch in willing-to-wait mode if you are already holding a higher level latch; this error makes the subsequent comments about the complexity involved in dropping and re-acquiring cache buffers chains latches irrelevant – you don’t have to drop the latch.
There is another odd error in the same sentence – I don’t say anything about the latch level and it’s use in controlling the order in which willing-to-wait gets can be made.

Relative
Uniquely identifies a datafile within a tablespace. For small and medium size databases, relative file numbers usually have the same value as the absolute file number. However, when the number of datafiles in a database exceeds a threshold (typically 1023), the relative file number differs from the absolute file number. In a bigfile tablespace, the relative file number is always 1024 (4096 on OS/390 platform).

The relative file number can’t uniquely identify a datafile, two datafiles in different tablespaces may have the same relative file number. We can produce such a case by event 10120. The datafiles of test1 and test1 hold the same relative file number 16.

==========================================
From $ORACLE_HOME/rdbms/mesg/oraus.msg
==========================================

10120, 00000, "generate relative file # different from absolute"
// *Cause:
// *Action:

sys@CS11GR2> select * from product_component_version;

PRODUCT                                  VERSION              STATUS
---------------------------------------- -------------------- --------------------
NLSRTL                                   11.2.0.2.0           Production
Oracle Database 11g Enterprise Edition   11.2.0.2.0           Production
PL/SQL                                   11.2.0.2.0           Production
TNS for Linux:                           11.2.0.2.0           Production

sys@CS11GR2> select platform_id, platform_name from v$database;

PLATFORM_ID PLATFORM_NAME
----------- -----------------
         10 Linux IA (32-bit)

sys@CS11GR2>
sys@CS11GR2> alter system set events '10120 trace name context forever, level 1';

System altered.

sys@CS11GR2>
sys@CS11GR2> create smallfile tablespace test1 logging datafile '/home/oracle/app/oracle/oradata/cs11gR2/test1.dbf'
			 size 5m extent management local segment space management auto;

Tablespace created.

sys@CS11GR2>
sys@CS11GR2> alter system set events '10120 trace name context off';

System altered.

sys@CS11GR2>
sys@CS11GR2> create smallfile tablespace test2 logging datafile '/home/oracle/app/oracle/oradata/cs11gR2/test2.dbf'
			 size 5m extent management local segment space management auto;

Tablespace created.

sys@CS11GR2>
sys@CS11GR2> select
  2    file#,
  3    rfile#,
  4    name
  5  from
  6    v$datafile
  7  where
  8    name like '%test_.dbf%';

     FILE#     RFILE# NAME
---------- ---------- ------------------------------
        15         16 /home/oracle/app/oracle/oradat
                      a/cs11gR2/test1.dbf

        16         16 /home/oracle/app/oracle/oradat
                      a/cs11gR2/test2.dbf

Two tables t1/t2 are created on the two tablespaces respectively, we can see the segment header have same block number. Data block dump on the two blocks show they have the same RDBA 0×04000082 (16/130). RDBA is 32 bit length, the format is: {relative_file_number[10]} {block number[22]}.

sys@CS11GR2> create table t1 tablespace test1 as select object_id, object_name from all_objects where rownum=1;

Table created.

sys@CS11GR2> create table t2 tablespace test2 as select object_id, object_name from all_objects where rownum=1;

Table created.

sys@CS11GR2>
sys@CS11GR2> col seg_owner head OWNER for a20
sys@CS11GR2> col seg_ts head SEGMENT_NAME for a30
sys@CS11GR2> col seg_segment_type head SEGMENT_TYPE for a20
sys@CS11GR2>
sys@CS11GR2> select
  2    owner seg_owner,
  3    segment_name seg_segment_name,
  4    segment_type seg_segment_type,
  5    tablespace_name seg_tablespace_name,
  6    blocks,
  7    header_file hdrfil,
  8    header_block hdrblk
  9  from
 10    dba_segments
 11  where
 12    segment_name in ('T1','T2');

OWNER SEGMENT_NAME  SEGMENT_TYPE SEG_TS  BLOCKS HDRFIL     HDRBLK
----- ------------- ------------ ------ ------- ------ ----------
SYS   T1            TABLE        TEST1        8     15        130
SYS   T2            TABLE        TEST2        8     16        130

sys@CS11GR2> @dump_blk 15 130
sys@CS11GR2> alter system dump datafile &1 block &2;

System altered.

sys@CS11GR2> @dump_blk 16 130
sys@CS11GR2> alter system dump datafile &1 block &2;

System altered.

Start dump data blocks tsn: 16 file#:15 minblk 130 maxblk 130
Block dump from cache:
Dump of buffer cache at level 4 for tsn=16, rdba=67108994
BH (0x253ec0b4) file#: 15 rdba: 0x04000082 (16/130) class: 4 ba: 0x25186000
  set: 5 pool: 3 bsz: 8192 bsi: 0 sflg: 1 pwc: 0,28
  dbwrid: 0 obj: 91002 objn: 91002 tsn: 16 afn: 15 hint: f
  hash: [0x233f5db4,0x3990fdf8] lru: [0x223e9300,0x21ffb8fc]
  obj-flags: object_ckpt_list
  ckptq: [0x21ffb850,0x223e9254] fileq: [0x21ffb858,0x223e925c] objq: [0x324359f0,0x21ffb914] objaq: [0x324359e0,0x21ffb91c]
  st: XCURRENT md: NULL tch: 1
  flags: buffer_dirty redo_since_read
  LRBA: [0x3a5.4a2ba.0] LSCN: [0x1.cf38a331] HSCN: [0x1.cf38a33e] HSUB: [2]
  cr pin refcnt: 0 sh pin refcnt: 0

Start dump data blocks tsn: 17 file#:16 minblk 130 maxblk 130
Block dump from cache:
Dump of buffer cache at level 4 for tsn=17, rdba=67108994
BH (0x223f67fc) file#: 16 rdba: 0x04000082 (16/130) class: 4 ba: 0x2231a000
  set: 6 pool: 3 bsz: 8192 bsi: 0 sflg: 1 pwc: 0,28
  dbwrid: 0 obj: 91003 objn: 91003 tsn: 17 afn: 16 hint: f
  hash: [0x27bf5ce0,0x3990fe00] lru: [0x237e4af4,0x277e282c]
  obj-flags: object_ckpt_list
  ckptq: [0x277e2780,0x237e4a48] fileq: [0x277e2788,0x237e4a50] objq: [0x32434ca8,0x277e2844] objaq: [0x32434c98,0x277e284c]
  st: XCURRENT md: NULL tch: 1
  flags: buffer_dirty redo_since_read
  LRBA: [0x3a5.4a2f2.0] LSCN: [0x1.cf38a357] HSCN: [0x1.cf38a35f] HSUB: [2]
  cr pin refcnt: 0 sh pin refcnt: 0

To distinct the two block, we need something else, such as data_object_id, absolute file number or tablespace number. An example is the rowid.

sys@CS11GR2> select
  2    rowid
  3  from
  4    t1;

ROWID
------------------
AAAWN6AAQAAAACDAAA

sys@CS11GR2>
sys@CS11GR2> select
  2    rowid
  3  from
  4    t2;

ROWID
------------------
AAAWN7AAQAAAACDAAA

The format of the rowid is: {data_object_id[6]} {relative_file_number[3]} {block number[6]} {rownumber[3]}. In this case, the relative file number of both rowid is 16. AAA is 0, ascii(‘Q’) – ascii(‘A’) = 16, so AAQ is 16. In such a case, oracle use the data_object_id(AAAWN6) to fetch the segment_name, absolute file number and tablespace number.

推荐Gary Millsap近期两篇相关的论文
Mastering Performance with Extended SQL Trace
Thinking Clearly About Performance – Cary Millsap

Cost-Based Oracle Fundamentals – Jonathan Lewis
在Jonathan Lewis庖丁解牛之后, Oracle的优化器和SQL调优不再神秘.

Oracle性能诊断艺术 – Christian Antognini 译者: 童家旺,胡怡文,冯大辉
深入系统的阐述Oracle的各种调试方法和技巧, 它们各自的运用场景. 应该是DBA案头必备, 翻译得很好.

Expert Oracle Database Architecture – Thomas Kyte
Tom Kyte的经典,时常温故, 时常知新.

Scaling Oracle8i – James Morle
请忽略8i两个字,设计原则很少需要更改. 如何构建基于Oracle的良好扩展的应用, 本书是Oracle在类Unix操作系统上的实践论, 其中关于CPU/缓存/IO的硬件知识尤为难得.

Oracle8i Internal Services for Waits, Latches, Locks, and Memory – Steve Adams
关于Oracle内部原理的经典,一直以来是DBA津津乐道的对象. 内容密度很大, 每次翻看都有新的收获. 只是没有测试用例的辅助,门槛提高了很多. 在Jonathan Lewis的Oracle Core出版之后, 这本书应该会慢慢退出历史的舞台.

海量数据库解决方案 – (韩)李华植 译者:郑保卫,盖国强
国内少有的从韩国引进的数据库书籍, 本书以Oracle为基础, 阐述如何设计和优化基于关系数据库的系统. 其中索引的设计以及对各种连接方法的解释让人豁然开朗. 这本书可以拿来补充数据库理论.

Forecasting Oracle Performance – Shallahamer, Craig
利用各种数学模型预测性能走向, 以此判断系统是否需要扩容. 读得比较匆忙, 有些内容比较深入.

从11月8号到现在差不多1个半月的时间，我把这本书看了三遍。Jonathan Lewis擅长用简单明了的语言，描述Oracle复杂的内部机制，这种化繁为简的功力或许来自长期实战经验和研究的积累。精心设计的测试用例很大程度降低这本书的门槛，如果你想对oracle知其所以然，这是一本不应该错过的书。

以下是Tanel Poder对这本书评论中的一句，另外，Charles Hooper也有一篇详细的评论，全书错误列表可以在这里更新。

It will likely be the best Oracle internals book out there for the coming 10 years, just like Steve Adams’es Oracle Internal Services book was in the last decade

Oracle Core很可能是未来十年关于Oracle内部原理最好的书，就像过去十年我们对Steve Adams的Oracle Internal Services一直津津乐道一样

作者在第一章从一个简化的原理图出发，做了一个轻松的开始，后面7个章节的内容密度却让人惊叹，对我自己而言，读到第三遍之后，才把前后各个知识点联系起来。除了知识本身，Jonathan所展示的他研究Oracle的思路和方法或许更有借鉴意义。

第二章通过对data block, undo block和redo的转储，简单的说明Oracle最核心的机制redo和undo。那么Oracle如何利用redo和undo实现事物四个特性中的ACID中的A-Atomicity, C-Consistency, I-Isolation的呢？Oracle通过什么样的数据结构把transaction、undo和数据块上的ITL组织起来？你很可能从Tom Ktye的书里知道Oracle具备了multi versions, consistent read, current read的能力，那它们是如何实现的？Oracle如何fast commit, 为什么有commit cleanout和delayed block cleanout， 什么是transaction TABLES， transaction table也能回滚？ 为什么会出现ORA-01555这种错误呢？第三就解释了这些问题。

第四章， Oracle在多用户的情况下利用latch和Lock，保护数据不被破坏。不同的锁机制运用在什么不同的场景，背后的实现逻辑分别是怎样的？比如Latch的原子操作，enqueue的三个队列(owners,converters,waiters)，不同模式之间的兼容性。

第五章，从Oracle以Granule为单位管理内存，Buffer pool和Granule, Buffer pool和Working Data Set, Working data set和Granue之间是什么关系， Oracle如何用LRU+TCH的算法来有效管理缓存，LRU链表和hash链表、cache buffers chains和cache buffers lru chain这两种latch如何互动。Oracle里如何利用典型的哈希表和双向链表的结合，快速的查找已经缓存的数据。

第六章阐述Oracle 优先写log的方式来实现事物四个特性ACID中的D-Durability，保证事物持久性，通过各种checkpiont机制(incremetal checkpoint, media recovery checkpoint)，尽量缩短instance recovery和media recovery所需要的时间，同时最大限度的减小对性能的影响。Log writter和database如何工作，他们之间如何沟通。

第七章关于在Oracle里，SQL被解析，优化和执行。软解析和硬解析的过程是怎样的？为什么有些情况下可以不出现解析？为什么在OLTP中使用绑定变量可以很大程度减少latch的争用? Ditionary Cache和library cache的组织结构，shared pool和buffer pool内存管理方式有什么异同？如何应对ORA-04031错误？

第八章，RAC所标榜的高可用，扩展性如何实现的。RAC如何通过GRD(Global Resource Directory)在实例之间协调各种资源的使用，GCS和GES如何协调工作，Case Fusion的原理是怎样的，RAC的水平扩展如何实现。

最后的附录作者分享了如何通过各种转储和调试工具和方法，研究Oracle的内部原理。授人以鱼不如授人以渔。

-------------------------------------------------------------------------------------------------------------
| Id  | Operation                      | Name             | Starts | A-Rows |   A-Time   | Buffers | Reads  |
-------------------------------------------------------------------------------------------------------------
|   1 |  HASH UNIQUE                   |                  |      1 |      0 |00:00:02.91 |    3829 |    387 |
|*  2 |   TABLE ACCESS BY INDEX ROWID  | BBC_OR_RTY       |      1 |      0 |00:00:02.91 |    3829 |    387 |
|   3 |    NESTED LOOPS                |                  |      1 |    764 |00:00:04.19 |    3829 |    387 |
|   4 |     TABLE ACCESS BY INDEX ROWID| BBC_OR_EVER      |      1 |    763 |00:00:00.01 |     777 |      0 |
|*  5 |      INDEX SKIP SCAN           | BBC_OR_EVER_IDX4 |      1 |    763 |00:00:00.01 |      19 |      0 |
|*  6 |     INDEX RANGE SCAN           | RC_ORPARTY_IDX01 |    763 |      0 |00:00:02.90 |    3052 |    387 |
-------------------------------------------------------------------------------------------------------------

To know where comes the buffer gets, I first enabled the extended sql trace for the session. The trace file shows the session are doing lots of phsical read – “db file sequential read”. During the 45 minute tracing range, over 90% phsical read is on the data file 152. The obj# is 0, which give me a hint that the reads are for undo data. It turns out The data file with number 152 is undo datafile. Most of the undo phisical reads is following the phsical read on the object index BBC_OR_EVER_IDX4(obj#=193624), It means that undo records need to be applied for creating CR version on the index blocks.

WAIT #1: nam='db file sequential read' ela= 8999 file#=190 block#=501237 blocks=1 obj#=193624
WAIT #1: nam='db file sequential read' ela= 6331 file#=152 block#=2608161 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 15672 file#=152 block#=2608160 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 7966 file#=152 block#=2608155 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 14522 file#=152 block#=2608159 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 662 file#=152 block#=2608158 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 221 file#=152 block#=2608156 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 19045 file#=152 block#=2608153 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 6853 file#=152 block#=2608150 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 23853 file#=152 block#=2608145 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 14253 file#=152 block#=2608139 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 3329 file#=152 block#=2608136 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 6137 file#=152 block#=2608131 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 10516 file#=152 block#=2608115 blocks=1 obj#=0
WAIT #1: nam='db file sequential read' ela= 8209 file#=152 block#=2608125 blocks=1 obj#=0

oracle@appserver:~/trace$ grep "file#=152" bbcdb1_ora_29189.trc | wc -l
357935
oracle@appserver:~/trace$ grep "file#=" bbcdb1_ora_29189.trc | wc -l
389577

After capturing the session statistics, I found some interesting figure. The session have done 8764402 consistent gets, 8338399 of 8764402 gets are on undo data. The session need to rollback transaction tables 446 times, which acquire 8040580 undo records to apply. On average, 18028 undo records need to be applied to rollback a segment header block until the sesison can identify the correct SCN. The session did not fail on ORA-01555 means through great effort, it can find the proper SCN everytime.
The index blocks of BBC_OR_EVER_IDX4 is not cleanout at the moment they were written to disk. When the following sessions access the index block, it will found here is active transaction amond the intested transaction list, following the XID in ITL slot, the session will go to the corresponding undo segment header, apply undo records and rollback the transaction table, to identify when the transaction was commit;

Name                                                       Value
---------------------------------------------------------  -------
session logical reads                                      8775885
consistent gets                                            8764402
consistent gets - examination                              8338399
consistent changes                                         8041388
transaction tables consistent reads - undo records applied 8040580
transaction tables consistent read rollbacks               446
data blocks consistent reads - undo records applied        386

To fixed the problem, I need to issue a manually cleanout, that’s a fast full index scan.

select /*+index_ffs(or bbc_or_ever_idx4)*/ count(*) from bbc_or_ever or where last_mod_dt > sysdate - 91;

[oracle@appsever fd]$ lsnrctl

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 05-JAN-2012 15:28:00

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> show log_directory
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER parameter "log_directory" set to /home/oracle/RAC/10.2.0/network/log/
The command completed successfully
LSNRCTL> sho log_file
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER parameter "log_file" set to listener_appsever.log
The command completed successfully
LSNRCTL> exit

[oracle@appsever fd]$ cd /home/oracle/RAC/10.2.0/network/log/
[oracle@appsever log]$ ls -l listener_appsever.log
-rw-r--r--  1 oracle oinstall 537774814 Jan  5 15:29 listener_appsever.log
[oracle@appsever log]$ du -sh
521M    .
[oracle@appsever log]$ rm listener_appsever.log
[oracle@appsever log]$ ls -l listener_appsever.log
ls: listener_appsever.log: No such file or directory
[oracle@appsever log]$ du -h
7.2M    .
[oracle@appsever log]$ ps -ef | grep lsnr
oracle   13126  9110  0 15:30 pts/0    00:00:00 grep lsnr
oracle   30017     1  0  2011 ?        00:00:10 /home/oracle/RAC/10.2.0/bin/tnslsnr LISTENER_appsever -inherit
[oracle@appsever log]$ cd /proc/30017/fd
[oracle@appsever fd]$ ls -l
total 23
lrwx------  1 oracle oinstall 64 Jan  5 15:19 0 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 1 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 10 -> socket:[62869]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 11 -> socket:[62870]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 12 -> socket:[62872]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 13 -> socket:[63158]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 14 -> socket:[7596135]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 15 -> socket:[7596175]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 16 -> socket:[63235]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 17 -> socket:[67502]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 18 -> socket:[68706]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 19 -> socket:[68708]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 2 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 20 -> socket:[69906]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 21 -> socket:[69909]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 22 -> socket:[70880]
l-wx------  1 oracle oinstall 64 Jan  5 15:19 3 -> /home/oracle/RAC/10.2.0/network/log/listener_appsever.log (deleted)
lr-x------  1 oracle oinstall 64 Jan  5 15:19 4 -> pipe:[62841]
lr-x------  1 oracle oinstall 64 Jan  5 15:19 5 -> /home/oracle/RAC/10.2.0/network/mesg/nlus.msb
lr-x------  1 oracle oinstall 64 Jan  5 15:19 6 -> /home/oracle/RAC/10.2.0/network/mesg/tnsus.msb
l-wx------  1 oracle oinstall 64 Jan  5 15:19 7 -> pipe:[62842]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 8 -> socket:[62866]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 9 -> socket:[62867]

If you copy back the file descriptor to the log directory, you can see there are updated entries in the log file. The file size has increase from 537774814 to 537775803.

[oracle@appsever fd]$ cp 3 /home/oracle/RAC/10.2.0/network/log/listener_appsever.log
[oracle@appsever fd]$ cd -
/home/oracle/RAC/10.2.0/network/log
[oracle@appsever log]$ ls -l listener_appsever.log
-rw-r--r--  1 oracle oinstall 537775803 Jan  5 15:44 listener_appsever.log

There is a set log_file command in lsnrctl tool, to let the listener new a log file and release the previous one, so you can safely archive the old log file. Here comes a little demo.

[oracle@appsever log]$ lsnrctl

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 05-JAN-2012 15:47:08

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set log_file listener_appsever_2012_JAN_05.log
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER parameter "log_file" set to listener_appsever_2012_jan_05.log
The command completed successfully
LSNRCTL> exit
[oracle@appsever log]$ ls -ltr
total 532968
-rw-r--r--  1 oracle oinstall         0 Jan  4  2010 listener.log
-rw-r-----  1 oracle oinstall         8 Mar 10  2010 listener_appsever.log.size
-rw-r--r--  1 oracle oinstall   7426228 Jan  5 15:33 sqlnet.log
-rw-r--r--  1 oracle oinstall 537775803 Jan  5 15:44 listener_appsever.log
-rw-r--r--  1 oracle oinstall        36 Jan  5 15:47 listener_appsever_2012_jan_05.log
[oracle@appsever log]$ cd -
/proc/30017/fd
[oracle@appsever fd]$ ls -l
total 23
lrwx------  1 oracle oinstall 64 Jan  5 15:19 0 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 1 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 10 -> socket:[62869]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 11 -> socket:[62870]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 12 -> socket:[62872]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 13 -> socket:[63158]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 14 -> socket:[7596135]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 15 -> socket:[7596175]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 16 -> socket:[63235]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 17 -> socket:[67502]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 18 -> socket:[68706]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 19 -> socket:[68708]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 2 -> /dev/null
lrwx------  1 oracle oinstall 64 Jan  5 15:19 20 -> socket:[69906]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 21 -> socket:[69909]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 22 -> socket:[70880]
l-wx------  1 oracle oinstall 64 Jan  5 15:19 3 -> /home/oracle/RAC/10.2.0/network/log/listener_appsever_2012_jan_05.log
lr-x------  1 oracle oinstall 64 Jan  5 15:19 4 -> pipe:[62841]
lr-x------  1 oracle oinstall 64 Jan  5 15:19 5 -> /home/oracle/RAC/10.2.0/network/mesg/nlus.msb
lr-x------  1 oracle oinstall 64 Jan  5 15:19 6 -> /home/oracle/RAC/10.2.0/network/mesg/tnsus.msb
l-wx------  1 oracle oinstall 64 Jan  5 15:19 7 -> pipe:[62842]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 8 -> socket:[62866]
lrwx------  1 oracle oinstall 64 Jan  5 15:19 9 -> socket:[62867]

1. Lock conversion
Lock conversion appears in the common foreign key problem, If supporting index is missing on a foreign key, and a session is try to update or delete on the parent table, it will lock the child table with mode 4. If the session already hold a lock on child table with mode 3, when it update or delete on the parent table, it’ll convert the lock to mode 5.

2. The priority of the converters and waiters
There are 3 queues per enqueue, owners/converters/waiters. Once the owner release the enqueue, which session will hold the lock first, the one in converters queue or the one in waiters queue?

The code to build the parent and child table, and there is supporting index on FK in child table.

oe@cs10g> drop table child;

table dropped.

oe@cs10g> drop table parent;

table dropped.

oe@cs10g>
oe@cs10g> create table parent(id number, vc varchar2(100));

table created.

oe@cs10g> alter table parent add constraint parent_pk primary key(id)
  2  using index (create unique index parent_pk on parent(id));

table altered.

oe@cs10g>
oe@cs10g> create table child(id number, pid number, vc varchar2(100));

table created.

oe@cs10g>
oe@cs10g> alter table child add constraint child_pk primary key(id)
  2  using index (create unique index child_pk on child(id));

table altered.

oe@cs10g>
oe@cs10g> alter table child add constraint child_fk
  2  foreign key(pid) references parent(id);

table altered.

oe@cs10g>
oe@cs10g> insert into parent values(1,lpad('1',100,'0'));

1 row created.

oe@cs10g> insert into parent values(2,lpad('2',100,'0'));

1 row created.

oe@cs10g> insert into parent values(3,lpad('3',100,'0'));

1 row created.

oe@cs10g> insert into parent values(4,lpad('4',100,'0'));

1 row created.

oe@cs10g>
oe@cs10g> commit;

commit complete.

oe@cs10g>
oe@cs10g> insert into child values(1,1,lpad('1',100,'0'));

1 row created.

oe@cs10g> insert into child values(2,2,lpad('2',100,'0'));

1 row created.

oe@cs10g> insert into child values(3,3,lpad('3',100,'0'));

1 row created.

oe@cs10g>
oe@cs10g> exec dbms_stats.gather_table_stats(user,'parent');

pl/sql procedure successfully completed.

oe@cs10g> exec dbms_stats.gather_table_stats(user,'child');

pl/sql procedure successfully completed.

oe@cs10g>
oe@cs10g> select object_id from user_objects where object_name='child';

 object_id
----------
    535797

Sequence of actions is as below:
session 141, delete the only child of parent 1
session 142, delete the only child of parent 2
session 144, Attempt to lock the child table in exclusive mode (and start to wait)
session 142, Attempt to delete parent 1 (and start to wait due to missing FK index)
session 152, Attemp to delete the only child of parent 3 (and start to wait)

From query on v$lock, we can see the session 142/144/152 is blocked. Session 142 hold the lock in mode 3 together with 141. When attempt to delete the parent 1, session 142 need to convert the lock mode from 3 to 5, but session already hold the lock in mode 3, mode 3 and 5 are not comptible, so session 142 is now blocked in converters queue. We can dump the enqueue info to confirm.

--session 141
19:18:15 oe@cs10g> delete from child where pid=1;

1 row deleted.

--session 142
19:18:28 oe@cs10g> delete from child where pid=2;

1 row deleted.

--session 144
19:18:49 oe@CS10G> lock table child in exclusive mode;

--session 142
19:19:16 oe@CS10G> delete from parent where id=1;

--session 152
19:19:42 oe@CS10G> delete from child where pid=3;

--the owners/converters/waiters on the enqueue TM-53579-0
19:20:16 oe@CS10G> select sid, type, id1, id2, lmode, request, ctime, block from v$lock where type='TM' and id1=53579;

       SID TY	     ID1	ID2	 LMODE	  REQUEST      CTIME	  BLOCK
---------- -- ---------- ---------- ---------- ---------- ---------- ----------
       141 TM	   53579	  0	     3		0	 292	      1
       142 TM	   53579	  0	     3		5	 280	      1
       144 TM	   53579	  0	     0		6	 250	      0
       152 TM	   53579	  0	     0		3	 205	      0

19:28:07 oe@CS10G> alter session set events 'immediate trace name enqueues level 3';

Session altered.

--the output on the enqueue TM-535797-0
res      identification         NUL SS  SX  S   SSX X   md link
         owners              converters          waiters
-------------------------------------------------------------------------
0x18013e650 TM-0000d14b-00000000 U   0   0   2   0   0   0  8 [180153c60,180153c60]
         [17f0345d8,17f0345d8] [17f0347d8,17f0347d8] [17f0348d8,17f034ad8]
   lock     que owner    session        hold wait ser link
   ----------------------------------------------------------------------
   0x17f0345c8 OWN 0x18076ae00 0x18076ae00 (141)   SX NLCK  33 [18013e660,18013e660]
   0x17f0347c8 CON 0x18076c368 0x18076c368 (142)   SX  SSX   8 [18013e680,18013e680]
   0x17f0348c8 WAT 0x18076ee38 0x18076ee38 (144) NLCK    X  14 [17f034ad8,18013e670]
   0x17f034ac8 WAT 0x180779978 0x180779978 (152) NLCK   SX  25 [18013e670,17f0348d8]

Once sessin 141 commit the change, session 142 become the owner of the enqueue, although session 144 wait more time in the waiters queue. Because the session in the converters queues have higher priority.

19:31:37 oe@CS10G> select sid, type, id1, id2, lmode, request, ctime, block from v$lock where type='TM' and id1=53579;

       SID TY	     ID1	ID2	 LMODE	  REQUEST      CTIME	  BLOCK
---------- -- ---------- ---------- ---------- ---------- ---------- ----------
       142 TM	   53579	  0	     3		0	   3	      1
       144 TM	   53579	  0	     0		6	 763	      0
       152 TM	   53579	  0	     0		3	 718	      0

19:31:43 oe@CS10G> alter session set events 'immediate trace name enqueues level 3';

Session altered.

--the output on the enqueue TM-535797-0
res      identification         NUL SS  SX  S   SSX X   md link
         owners              converters          waiters
-------------------------------------------------------------------------
0x18013e650 TM-0000d14b-00000000 U   0   0   1   0   0   0  8 [180153c60,180153c60]
         [17f0347d8,17f0347d8] [18013e680,18013e680] [17f0348d8,17f034ad8]
   lock     que owner    session        hold wait ser link
   ----------------------------------------------------------------------
   0x17f0347c8 OWN 0x18076c368 0x18076c368 (142)   SX NLCK   8 [18013e660,18013e660]
   0x17f0348c8 WAT 0x18076ee38 0x18076ee38 (144) NLCK    X  14 [17f034ad8,18013e670]
   0x17f034ac8 WAT 0x180779978 0x180779978 (152) NLCK   SX  25 [18013e670,17f0348d8]

First come first service, when session 142 commit, session 144 with greater ctime get the enqueue. Be noted when a session hold the enqueue, the ctime will be reset to 0.

19:31:45 oe@CS10G> select sid, type, id1, id2, lmode, request, ctime, block from v$lock where type='TM' and id1=53579;

       SID TY	     ID1	ID2	 LMODE	  REQUEST      CTIME	  BLOCK
---------- -- ---------- ---------- ---------- ---------- ---------- ----------
       144 TM	   53579	  0	     6		0	   5	      1
       152 TM	   53579	  0	     0		3	 811	      0

19:33:14 oe@CS10G> alter session set events 'immediate trace name enqueues level 3';

Session altered.

--the output on the enqueue TM-535797-0
res      identification         NUL SS  SX  S   SSX X   md link
         owners              converters          waiters
-------------------------------------------------------------------------
0x18013e650 TM-0000d14b-00000000 U   0   0   0   0   0   1 40 [180153c60,180153c60]
         [17f0348d8,17f0348d8] [18013e680,18013e680] [17f034ad8,17f034ad8]
   lock     que owner    session        hold wait ser link
   ----------------------------------------------------------------------
   0x17f0348c8 OWN 0x18076ee38 0x18076ee38 (144)    X NLCK  14 [18013e660,18013e660]
   0x17f034ac8 WAT 0x180779978 0x180779978 (152) NLCK   SX  25 [18013e670,18013e670]

You may still wonder why the session can find the enqueue and attached itself to the owners/converts/waiters queue properly? Or what happen if 141 attempt to delete the parent 2, rather than than commit? Jonathan Lewis explain such detail in his book.

oe@CS11GR2> drop table t;

Table dropped.

oe@CS11GR2> create table t (x int, y int);

Table created.

oe@CS11GR2> insert into t values(1, 1);

1 row created.

oe@CS11GR2> commit;

Commit complete.
oe@CS11GR2> create or replace trigger t_bufer
  2  before update on t for each row
  3  begin
  4  	     dbms_output.put_line
  5  	     ( 'old.x = ' || : old.x ||
  6  	       ', old.y = ' || : old.y );
  7  	     dbms_output.put_line
  8  	     ( 'new.x = ' || :new.x ||
  9  	       ', new.y = ' || :new.y );
 10  end;
 11  /
Trigger created.

--session 1
oe@CS11GR2>
oe@CS11GR2> update t set x = x + 1;
old.x = 1, old.y = 1
new.x = 2, new.y = 1

1 row updated.

In session 2, run another update, with column x in where clause, it hange because the only row is locked by session 1. If I commit in session 1, the value of x is changed to 2, the update in session 2 need to restart, the trigger will fire twice.

--session 2, trigger fire twice if session 1 commit
oe@CS11GR2> update t set x = x + 1 where x > 0;
old.x = 1, old.y = 1
new.x = 2, new.y = 1
old.x = 2, old.y = 1
new.x = 3, new.y = 1

1 row updated.

If I rollback in session 1. there is no change on x, the update in session 2 don’t need to restart, the trigger will just fire once.

--session 2, trigger fire once if session 1 rollback
oe@CS11GR2> update t set x = x + 1 where x > 0;
old.x = 1, old.y = 1
new.x = 2, new.y = 1

1 row updated.

Now I want to check the statistics on session 2, so I take a snapshot of the statistics before and after the update. If session 1 commit the update, there is a “user commits” although the update in session 2 fail and restart implicitly; if session 1 rollback, there is one “user rollbacks”, although the update in session 2 does not restart. If I enable sql trace for session 2, there is no commit record. If I dump redo log in both case, there is no commit(OP Code 5.4) record for session 2 in both case also. When session 1 commit, session 2 have two redo change of transaction start marker(OP Code 5.2), the second marker reuse the previous transaction slot, without advanced the wrap#(sqn: 0×00000000). When session 1 rollback, there no repeat redo change (OP Code 5.2). The redo dump result is as expected.
Actually, the statistics of session 2 just look like session 1. Session 1 commit, session 2 have a “user commits”, session 1 rollback, session 2 have a “user rollbacks” and a “transaction rollbacks”. In this case, both the “user commits”, “user rollbacks” and “transaction rollbacks” seems false. That’s why I think the statistics of “user commits” and “user rollbacks” in session 2 are interesting, the oddity may be related to the cleanout mechanism and I hope someone will explain this
My testing is on 10.2.0.4 on OS X, and 11.2.0.3 on Linux, below output are from 11g.
The statistics of session 2, when session 1 commit, session 2 have two redo change of transaction start marker(OP Code 5.2), the second marker reuse the previous transaction slot, without advanced the wrap#(sqn: 0×00000000)

Name                                                   Value
----                                                   -----
user commits                                               1
db block gets                                             13
consistent gets                                           35
consistent gets - examination                              7
db block changes                                          13
consistent changes                                         4
redo entries                                               7
redo size                                              1,740
redo synch time (usec)                                   764
redo synch writes                                          1
data blocks consistent reads - undo records applied        4
cleanouts and rollbacks - consistent read gets             2
rollback changes - undo records applied                    1

REDO RECORD - Thread:1 RBA: 0x00037d.00000005.0010 LEN: 0x0200 VLD: 0x05
SCN: 0x0001.ceea5bc4 SUBSCN:  1 12/29/2011 15:38:30
(LWN RBA: 0x00037d.00000005.0010 LEN: 0003 NST: 04d SCN: 0x0001.ceea5bc4)
CHANGE #1 TYP:0 CLS:29 AFN:3 DBA:0x00c000e0 OBJ:4294967295 SCN:0x0001.ceea58eb SEQ:1 OP:5.2 ENC:0 RBL:0
ktudh redo: slt: 0x0010 sqn: 0x000014e8 flg: 0x0012 siz: 160 fbi: 0
            uba: 0x00c12a6e.0c5c.10    pxid:  0x0000.000.00000000

REDO RECORD - Thread:1 RBA: 0x00037d.00000006.00e4 LEN: 0x0138 VLD: 0x01
SCN: 0x0001.ceea5bc5 SUBSCN:  1 12/29/2011 15:38:30
CHANGE #1 TYP:0 CLS:29 AFN:3 DBA:0x00c000e0 OBJ:4294967295 SCN:0x0001.ceea5bc4 SEQ:2 OP:5.2 ENC:0 RBL:0
ktudh redo: slt: 0x0010 sqn: 0x00000000 flg: 0x0002 siz: 108 fbi: 1
            uba: 0x00c12a6e.0c5c.11    pxid:  0x0000.000.00000000

The statistics of session 2, when session 1 rollback

Name                                                   Value
----                                                   -----
user rollbacks                                             1
db block gets                                              9
db block gets from cache                                   9
consistent gets                                           23
consistent gets - examination                              8
db block changes                                          12
consistent changes                                         4
redo entries                                               8
redo size                                              1,636
redo write time                                            1
redo synch time (usec)                                   753
redo synch writes                                          1
data blocks consistent reads - undo records applied        4
cleanouts and rollbacks - consistent read gets             2
rollback changes - undo records applied                    2
transaction rollbacks                                      1

The code to capture statistics of session 2 and dump redo log

alter system switch logfile;
exec snap_stats.start_snap;
update t set x = x + 1 where x > 0;
exec snap_stats.end_snap;
prompt  =====================================
prompt  Sleeping 7 sescnds for redo log flush
prompt  =====================================
exec dbms_lock.sleep(7);
exec dump_log;

Update(first day in 2012):
Is the “user commits” and “user rollbacks” are really false?
No, the mysterious commits or rollback come from the session 1. Why? I capture the whole system statistics not my session statstics. I should have use the snap_my_stats not snap_stats. What a shame!
There is no “user commits” or “user rollbacks” in session 2 actually, no matter session 1 commit or rollback the update. Here comes the correct statistics. The previous redo dump and description is still correct anyway:-)

The statistics of session 2, when session 1 commit.

---------------------------------
Session stats - 01-Jan 15:26:44
Interval:-  16 seconds
---------------------------------
Name									 				Value
----									 				-----
db block gets								     		6
consistent gets 							    		18
consistent gets - examination						    5
CR blocks created							     		2
redo entries								     		4
redo size								   				908
data blocks consistent reads - undo records applied	    2
cleanouts and rollbacks - consistent read gets		    2
immediate (CR) block cleanout applications			    2
deferred (CURRENT) block cleanout applications			2

the statistics of session 2, when session 1 rollback.

---------------------------------
Session stats - 01-Jan 15:28:48
Interval:-  21 seconds
---------------------------------
Name									 				Value
----									 				-----
db block gets								     		4
consistent gets 							    		12
consistent gets - examination						    5
CR blocks created							     		2
redo entries								     		2
redo size								   				172
data blocks consistent reads - undo records applied	    2
cleanouts and rollbacks - consistent read gets		    2
immediate (CR) block cleanout applications			    2