[펌] 솔라리스 시스템에서 갑작스런 패닉 현상

 

솔라리스 시스템에서 갑작스런 패닉 현상이 나타나게 되면 보통 crash 디렉터리를 지정을

하지 않았다면 /var/crash/[hostnam]/ 밑에 unix 파일과 core 파일을 남깁니다..

그 파일을 이용하여 시스템 패닉전에 무슨일이 있었는지 분석이 가능 합니다.

아래는 분석에 사용하는 명령어들과 보기입니다.

우선 core 파일과 unix파일이 있는 디렉터리로 이동 합니다. ( /var/crash/[hostname] )

1. strings

# strings vmcore.0 | grep SunOS ---> OS 의 종류를 보여준다.

# strings vmcore.0 | grep macine ---> 어떤 호스트인지 열거 하여 준다.

# strings vmcore.0 | more ---> 다 보여준다.

2. netstat

# netstat -d unix.0 vmcore.0 --> 네트웍 통계보기

# netstat -n unix.0 vmcore.0 --> NFS통계보기

3. arp

# arp -a unix.0 vmcore.0 --> arp 테이블보기

4. ipcs

# ipcs -a -N unix.0 -C vmcore.0 --> ipc 보기

5. crash

사용법 : crash [ -d dumpfile ] [ -n namelist ] [ -w output-file ]

예)

# crash -d unix.0 vmcore.0

dumpfile = vmcore.0, namelist = /dev/ksyms, outfile = stdout

> status

system name: SunOS

release: 5.7

node name: ns

version: Generic_106541-15

machine name: sun4u

time of crash: 수 11월 7 11:23:10 2001

age of system: 1 hr., 6 min.

panicstr: [AFT1] errID 0x000003a4.70f365cc %sError(s)

See previous message(s) for details

panic registers:

pc: 10010104 sp: 400275d8

> help

help [-w filename] function[s]

help function

alias:

acceptable aliases are uniquely identifiable initial substrings

> help p

p [-e] [-f] [-l] [-w filename] [([-p] [-a] tbl_entry | #procid)... | -r]

tbl_entry = slot number | address | symbol | expression | range

process table

alias: proc

acceptable aliases are uniquely identifiable initial substrings

p -e

PROC TABLE SIZE = 2922

SLOT ST PID PPID PGID SID UID PRI NAME FLAGS

0 t 0 0 0 0 0 96 sched load sys lock

1 s 1 0 0 0 0 58 init load

2 s 2 0 0 0 0 98 pageout load sys lock nowait

3 s 3 0 0 0 0 60 fsflush load sys lock nowait

4 s 347 1 347 347 0 58 sac load jctl

5 s 350 1 350 350 0 37 dmispd load

6 s 138 1 138 138 0 41 keyserv load

7 s 53 1 53 53 0 43 devfseventd load

8 s 57 1 57 57 0 50 devfsadm load

9 s 136 1 136 136 0 58 rpcbind load

10 s 205 1 205 205 0 53 nscd load

11 s 130 1 130 130 60001 58 tocsin load

12 s 188 1 188 188 0 52 syslogd load

13 s 178 1 178 178 0 58 automountd load

14 s 170 1 170 170 0 58 in.named load jctl

15 s 163 1 163 163 0 0 inetd load

16 s 233 228 228 228 1006 23 mshttpd load

17 s 300 267 0 0 0 58 mysqld load

18 s 193 1 193 193 0 49 cron load

19 s 232 228 228 228 1006 33 mshttpd load

20 s 228 1 228 228 0 59 mshttpd load

21 s 234 228 228 228 1006 13 mshttpd load

22 s 235 228 228 228 1006 3 mshttpd load

23 s 236 228 228 228 1006 0 mshttpd load

24 s 249 1 249 249 0 58 utmpd load

25 s 267 1 0 0 0 10 safe_mysqld load

26 s 253 1 253 253 0 58 sendmail load jctl

27 s 265 1 265 265 0 58 httpd load

28 s 10139 265 265 265 60001 58 httpd load

29 s 10146 265 265 265 60001 58 httpd load

30 s 10141 265 265 265 60001 58 httpd load

31 s 10151 265 265 265 60001 58 httpd load

32 s 10140 265 265 265 60001 58 httpd load

33 s 281 1 279 279 0 60 auditd load

36 s 331 1 331 331 0 48 dtlogin load jctl

37 s 325 1 325 325 0 23 mountd load

38 s 327 1 327 327 0 33 nfsd load

40 s 351 347 347 347 0 58 listen load nowait jctl

41 s 348 1 348 348 0 55 ttymon load

42 s 14566 331 14566 14566 0 59 Xsun load

43 s 352 347 347 347 0 58 ttymon load jctl

44 s 10137 265 265 265 60001 53 httpd load

50 s 14585 1 14574 14574 0 59 fbconsole load

55 s 14605 14574 14605 14605 0 59 dtgreet load

60 s 14574 331 14574 14574 0 50 dtlogin load

71 s 10150 265 265 265 60001 58 httpd load

76 s 10138 265 265 265 60001 58 httpd load

82 s 11077 265 265 265 60001 58 httpd load

90 s 11085 265 265 265 60001 58 httpd load

> p -l

p_lock: owner 0 waiters 0

cr_lock: owner 0 waiters 0

Condition variable p_cv: 0

Condition variable p_flag_cv: 0

Condition variable p_lwpexit: 0

Condition variable p_holdlwps: 0

Condition variable p_srwchan_cv: 0

82 s 11077 265 265 265 60001 58 httpd load

p_lock: owner 0 waiters 0

cr_lock: owner 0 waiters 0

Condition variable p_cv: 0

Condition variable p_flag_cv: 0

Condition variable p_lwpexit: 0

Condition variable p_holdlwps: 0

Condition variable p_srwchan_cv: 0

90 s 11085 265 265 265 60001 58 httpd load

.

.

.

. 이하생략..

> q

6. adb 사용하기

# adb -k unix.0 vmcore.0 ( unix파일이나 core파일 대신 커널쪽을 볼수도 있다 ex> adb -k /dev/ksyms /dev/mem )

adb -k unix.0 vmcore.0

physmem 5b01

*panicstr/s ---> 패닉 메세지가 무엇이었나 보여준다.

0x40027768: [AFT1] errID 0x000003a4.70f365cc %sError(s)

See previous message(s) for details

$<utsname ---> 호스트 이름과 장비에 관해 열거해준다.

utsname:

utsname: sys SunOS

utsname+0x101: node ns

utsname+0x202: release 5.7

utsname+0x303: version Generic_106541-15

utsname+0x404: machine sun4u

srpc_domain/s ---> 도메인

srpc_domain:

srpc_domain:

hw_provider/s ---> 제조업체

hw_provider:

hw_provider: Sun_Microsystems

*time-(lbolt%0t100)=Y --->부트시간/날짜

2001 Oct 6 21:26:01

$<msgbuf ---> 최근 메세지 버펄를 열거 하여 준다

SunOS Release 5.7 Version Generic_106541-15 [UNIX(R) System V R

elease 4.0]

0x700b57e0: Copyright (c) 1983-1999, Sun Microsystems, Inc.

0x700b53c3: Ethernet address = 8:0:20:86:23:9e

0x700b4fa0: mem = 196608K (0xc000000)

0x700b4b80: avail mem = 188989440

0x700b4763: root nexus = Sun Ultra 1 SBus (UltraSPARC 167MHz)

0x700b4343: sbus0 at root: UPA 0x1f 0x0 ...

0x70155ee3: sbus0 is /sbus@1f,0

0x70155ac0: dma0 at sbus0: SBus0 slot 0xe offset 0x8400000

0x701556a3: dma0 is /sbus@1f,0/espdma@e,8400000

0x70155280: /sbus@1f,0/espdma@e,8400000/esp@e,8800000 (esp0):

esp-options=0x46

0x70154e60: esp0 at dma0: SBus0 slot 0xe offset 0x8800000 Onboard device spa

rc9 ipl 4

0x70154a43: esp0 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000

0x70154620: NOTICE: Memory scrubber exiting

0x70154200: sd0 at esp0:

0x70177d40: target 0 lun 0

0x70177923: sd0 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0

0x70177500: <FUJITSU-MAB3091SC-2703 cyl 7185 alt 2 hd 10 sec 248>

0x701770e0: sd1 at esp0:

0x70176cc0: target 1 lun 0

0x701768a3: sd1 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@1,0

0x70176480: <SUN4.2G cyl 3880 alt 2 hd 16 sec 135>

0x70176060: sd4 at esp0:

0x70179c00: target 4 lun 0

0x701797e3: sd4 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@4,0

0x701793c0: <SUN4.2G cyl 3880 alt 2 hd 16 sec 135>

0x70178fa0: sd6 at esp0:

0x70178b80: target 6 lun 0

0x70178763: sd6 is /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@6,0

0x70178347: root on /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a fstyp

e ufs

0x700efee0: zs0 at sbus0: SBus0 slot 0xf offset 0x1100000 Onboard device spa

rc9 ipl 12

0x700efac3: zs0 is /sbus@1f,0/zs@f,1100000

0x700ef6a0: zs1 at sbus0: SBus0 slot 0xf offset 0x1000000 Onboard device spa

rc9 ipl 12

0x700ef283: zs1 is /sbus@1f,0/zs@f,1000000

0x700eee66: keyboard is </sbus@1f,0/zs@f,1000000> major <29> minor <2>

0x700eea46: mouse is </sbus@1f,0/zs@f,1000000:b> major <29> minor <3>

0x700ee626: stdin is </sbus@1f,0/zs@f,1000000> major <29> minor <2>

0x700ee200: cgsix0 at sbus0: SBus0 slot 0x1 offset 0x0 SBus level 5 sparc9 i

pl 9

0x7015fea3: cgsix0 is /sbus@1f,0/cgsix@1,0

0x7015fbe1: cgsix0: screen 1152x900, single buffered, 1M mappable, rev 11

0x7015f926: stdout is </sbus@1f,0/cgsix@1,0> major <39> minor <0>

0x7015f660: cpu0: SUNW,UltraSPARC (upaid 0 impl 0x10 ver 0x22 clock 167 MHz)

0x7015f3a0: ledma0 at sbus0: SBus0 slot 0xe offset 0x8400010

0x7015f0e0: le0 at ledma0: SBus0 slot 0xe offset 0x8c00000 Onboard device sp

arc9 ipl 6

0x7015ee23: le0 is /sbus@1f,0/ledma@e,8400010/le@e,8c00000

0x7015eb63: dump on /dev/dsk/c0t0d0s1 size 1000 MB

0x7015e8a2: pseudo-device: devinfo0

0x7015e5e3: devinfo0 is /pseudo/devinfo@0

0x7015e320: fd0 at sbus0: SBus0 slot 0xf offset 0x1400000 Onboard device spa

rc9 ipl 11

0x7015e063: fd0 is /sbus@1f,0/SUNW,fdtwo@f,1400000

0x70559d60: sbusmem0 at sbus0: SBus0 slot 0x0 offset 0x0

0x70559aa3: sbusmem0 is /sbus@1f,0/sbusmem@0,0

0x705597e0: sbusmem1 at sbus0: SBus0 slot 0x1 offset 0x0

0x70559523: sbusmem1 is /sbus@1f,0/sbusmem@1,0

0x70559260: sbusmem2 at sbus0: SBus0 slot 0x2 offset 0x0

0x70558fa3: sbusmem2 is /sbus@1f,0/sbusmem@2,0

0x70558ce0: sbusmem3 at sbus0: SBus0 slot 0x3 offset 0x0

0x70558a23: sbusmem3 is /sbus@1f,0/sbusmem@3,0

0x7042b70b: WARNING: [AFT1] EDP event on CPU0 Data access at TL=0, errID 0x0

00003a4.70f365cc

AFSR 0x00000000.80400002<PRIV,EDP> AFAR 0xffffffff.ffffffff

AFSR.PSYND 0x0002(Score 95) AFSR.ETS 0x00 Fault_PC 0x1018af18

UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

0x7055834b: [AFT2] errID 0x000003a4.70f365cc No error found in ecache (No fa

ult PA available)

0x70558080: panic[cpu0]/thread=40027e60:

0x70558b80: [AFT1] errID 0x000003a4.70f365cc EDP Error(s)

See previous message(s) for details

0x70154fc0:

0x7015f7c3: syncing file systems...

0x7015fd43: [4]

0x7015f243: [4]

0x7015f500: panic[cpu0]/thread=4003fe60:

0x700ef800: panic sync timeout

0x7015ea00:

0x700efc23: dumping to /dev/dsk/c0t0d0s1, offset 209780736

$c ---> C 스택 역추적

complete_panic(0xf,0x10437c00,0x40027e60,0x7011478a,0x0,0x10107880) + 24

do_panic(0x1,0x40027954,0x10107f70,0x0,0x0,0x400277bb) + 174

vcmn_err(0x3,0x40027768,0x40027954,0x7efefeff,0x81010100,0xff00) + 14

cpu_aflt_log(0x400278f8,0x400277bb,0x40027978,0x3,0x1010785c,0x10107880) + 4c0

rootfs$<bootobj ---> 루투장치는 어떤것인가

rootfs:

rootfs: fstype ufs^@^@^@^@^@^@^@^@^@^@^@^@^@

rootfs+0x10: name /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a^@^@^@

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@^@^@^@^@^@^@^@

rootfs+0x90: flags size vp

0 0 0

swapfile$<bootobj ---> 스왑장치는 어떤것인지 열거

swapfile:

swapfile: fstype ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

swapfile+0x10: name ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

^@^@^@^@

swapfile+0x90: flags size vp

0 0 0

$q ---> 종료