qileilove
          

          blog已經轉移至github,大家請訪問 http://qaseven.github.io/
          
			
          
				
					
	
          
		
          
			Oracle RAC Database 11.1.0.6監聽故障案例
		
          
		
          　晚上接到電話，客戶的一套核心Oracle RAC數據庫連接不上，連接時報無監聽程序，客戶的Oracle RAC版本為11.1.0.6，平臺為AIX 6.1.05，使用了IBM HACMP 5.5.0.8。
          　　當我遠程過去的時候，發現節點2已經沒有任何oracle用戶的進程，且concurrent的vg沒有激活，HACMP的服務也offline。
          　　另一個節點Oracle的實例是正常的，且有部分服務器進程依然在工作，但是本地監聽器出現了故障，導致新的連接無法連接到實例，通過crs_stat -t看到兩個實例的監聽也都是OFFLINE狀態。
          　　在節點上并沒有發現有LISTENER進程，且手動殺掉了所有的服務器進程，在oracle用戶下啟動監聽時收到以下的報錯：
          $ lsnrctl start listener_cdfy740a
          LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:09:09
          Copyright (c) 1991, 2007, Oracle.  All rights reserved.
          Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
          TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
          System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
          Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
          Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
          Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
          TNS-12542: TNS:address already in use
          TNS-12560: TNS:protocol adapter error
          TNS-00512: Address already in use
          IBM/AIX RISC System/6000 Error: 67: Address already in use
          Listener failed to start. See the error message(s) above...
          　　10.107.64.1是該節點的vip地址，下面是RAC環境的hosts配置：
          10.107.64.1     vip1
          10.107.64.2     vip2
          10.107.64.3     cdfy740a
          10.107.64.4     cdfy740b
          172.201.201.1   prv1
          172.201.201.2   prv2
          　　手動停掉該節點的nodeapps服務：
          　　cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps -n cdfy740a
          　　成功停止后，VIP在主機層面已經消失：
          cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
          en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
          inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
          inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
          inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
          inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
          inet6 ::1%1/0
          tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
          　再次啟動節點nodeapps服務：
          cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps -n cdfy740a
          CRS-1006: No more members to consider
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:07
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle.  All rights reserved.
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12542: TNS:address already in use
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00512: Address already in use
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:   IBM/AIX RISC System/6000 Error: 67: Address already in use
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listener failed to start. See the error message(s) above...
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:08
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle.  All rights reserved.
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip1)(PORT=1521)))
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:   IBM/AIX RISC System/6000 Error: 79: Connection refused
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:   IBM/AIX RISC System/6000 Error: 79: Connection refused
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.3)(PORT=1521)(IP=FIRST)))
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
          cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:   IBM/AIX RISC System/6000 Error: 79: Connection refused
          CRS-0215: Could not start resource 'ora.cdfy740a.LISTENER_CDFY740A.lsnr'.
          　　之前使用lsnrctl status listener_cdfy740a查看監聽器狀態時也收到Connection refused的錯誤。
          　　查看主機層面已經成功綁定了VIP地址：
          cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
          en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
          inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
          inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
          inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
          inet 10.107.64.1 netmask 0xffffff00 broadcast 10.107.64.255
          tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
          lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
          inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
          inet6 ::1%1/0
          tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
          　　再次嘗試手動啟動本地監聽器：
          cdfy740a@root[/]su - oracle
          $ lsnrctl start listener_cdfy740a
          LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:18:37
          Copyright (c) 1991, 2007, Oracle.  All rights reserved.
          Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
          TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
          System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
          Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
          Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
          Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
          TNS-12542: TNS:address already in use
          TNS-12560: TNS:protocol adapter error
          TNS-00512: Address already in use
          IBM/AIX RISC System/6000 Error: 67: Address already in use
          Listener failed to start. See the error message(s) above...
          　　啟動依然失敗。
          　　檢查監聽器配置文件：
          $ cat listener.ora
          # listener.ora.cdfy740a Network Configuration File: /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora.cdfy740a
          # Generated by Oracle configuration tools.
          LISTENER_CDFY740A =
          (DESCRIPTION_LIST =
          (DESCRIPTION =
          (ADDRESS = (PROTOCOL = TCP)(HOST = vip1)(PORT = 1521))
          (ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.1)(PORT = 1521)(IP = FIRST))
          (ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.3)(PORT = 1521)(IP = FIRST))
          )
          )
          　　在監聽配置文件中，vip1和10.107.64.1是兩個重復的地址，手動將10.107.64.1所在行去掉之后，監聽即可正常的啟動。
          　　之后恢復節點2的HACMP服務，Oracle RAC隨即恢復正常。
          　　另外，還發現客戶的監聽日志已經被填得很大，大概在1.6GB左右，過大的監聽日志文件也會導致監聽器不穩定，這里將兩個節點的監聽日志進行了重命名操作。
          　　《10g RAC監聽器配置文件listener.ora中的IP=FIRST》：http://blog.itpub.net/23135684/viewspace-715967/
          　　《IP=FIRST作用說明》：http://www.xifenfei.com/2713.html
           
          
		
          
			posted on 2014-11-27 12:00 順其自然EVO 閱讀(246) 評論(0)  編輯  收藏  所屬分類: 測試學習專欄 
		
          
	
          
	
	


	


          
	    
    




          








          

				
			
          
			
          
				
					
          
	
          
		
          <2014年11月>
          
	
          日一二三四五六
          2627282930311
          2345678
          9101112131415
          16171819202122
          23242526272829
          30123456
          

          

					

          導航
          

					

          統計
          
	
					
					

          常用鏈接
          


          留言簿(55)
          


		
          隨筆分類
          
		
				
			
	
		
          隨筆檔案
          
		
				
			
	
		
          文章分類
          
		
				
			
	
		
          文章檔案
          
		
				
			
	



          搜索
          



          最新評論
	
          

          
	
			
		

          


          閱讀排行榜
          



          評論排行榜
          


				
			
          			
			

	

    







          
        
	




主站蜘蛛池模板：
安泽县|
文昌市|
博兴县|
武鸣县|
云和县|
进贤县|
绵竹市|
花莲市|
中方县|
海南省|
峨边|
新竹市|
玛沁县|
禄劝|
康马县|
南投县|
商南县|
定陶县|
永新县|
桃园县|
钟山县|
鄂伦春自治旗|
常德市|
涞水县|
伽师县|
文昌市|
寻甸|
平邑县|
贡山|
调兵山市|
志丹县|
龙游县|
江陵县|
滦南县|
松阳县|
嵊泗县|
休宁县|
贵溪市|
白河县|
聂拉木县|
图片|