FreeBSD：サーバー入れ直した

OSの話

ZFSのストレージが微妙な事になったので
いったんまっさらにして作り直すことに。
せっかくだからある程度構築が終わったところでzfsのストレージをテストしてみた。

Hardware
[plain][plain]hp ML110G6
CPU：Intel Core i5-650 HT on
Memory：2Gx4 計8GB
PCI-E：Intel PRO/1000 PT Dual Port Server Adapter / Gigabit[/plain]

使用したHDDはこんな感じ
最新だけど早くは無い、低消費電力モデルだしね
めんどくさいAFTも採用していないモデルなので余計なことを考えなくても済む

[plain]NEKOSABA /# camcontrol devlist
<VB0160EAVEQ HPG0> at scbus0 target 0 lun 0 (pass0,ada0)
<Hitachi HDS5C3020ALA632 ML6OA580> at scbus1 target 0 lun 0 (pass1,ada1)
<Hitachi HDS5C3020ALA632 ML6OA580> at scbus2 target 0 lun 0 (pass2,ada2)
<Hitachi HDS5C3020ALA632 ML6OA5C0> at scbus3 target 0 lun 0 (pass3,ada3)
<Hitachi HDS5C3020ALA632 ML6OA5C0> at scbus4 target 0 lun 0 (pass4,ada4)
<INTEL SSDSA2VP020G2 2CV102M5> at scbus5 target 0 lun 0 (pass5,ada5)[/plain]

さっそくPoolをcreateしまつ

[plain]NEKOSABA /# zpool create zpool raidz ada1 ada2 ada3 ada4
NEKOSABA /# zpool status
pool: zfsboot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zfsboot ONLINE 0 0 0
ada0p3 ONLINE 0 0 0

errors: No known data errors

pool: zpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0

errors: No known data errors

NEKOSABA /# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zfsboot 3.80G 139G 399M legacy
zfsboot/usr 2.56G 139G 2.22G /usr
zfsboot/usr/home 84K 139G 84K /usr/home
zfsboot/usr/src 349M 139G 349M /usr/src
zfsboot/var 865M 139G 865M /var
zfsboot/var/log 164K 139G 164K /var/log
zpool 122K 5.34T 43.4K /zpool

NEKOSABA /# df
Filesystem Size Used Avail Capacity Mounted on
zfsboot 139G 399M 139G 0% /
devfs 1.0k 1.0k 0B 100% /dev
zfsboot/usr 141G 2.2G 139G 2% /usr
zfsboot/usr/home 139G 84k 139G 0% /usr/home
zfsboot/usr/src 139G 349M 139G 0% /usr/src
zfsboot/var 139G 864M 139G 1% /var
zfsboot/var/log 139G 163k 139G 0% /var/log
devfs 1.0k 1.0k 0B 100% /var/named/dev
172.16.11.2:/var 407G 18G 389G 4% /mnt/var
zpool 5.3T 43k 5.3T 0% /zpool [/plain]

完成。アレイの作成と解放が一瞬で終了するのはzfsのすごいところ。この間わずか数分。
zfsマンセー

ローカルでのスループットはこんな感じになった。

[plain]NEKOSABA /var/db# dd if=/dev/zero of=/zpool/test20G.dat bs=1m count=20480
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 79.025378 secs (271746077 bytes/sec)
NEKOSABA /var/db# dd if=/zpool/test20G.dat of=/dev/null bs=1m count=20480
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 61.492131 secs (349229017 bytes/sec)[/plain]

iSCSIを使ってみようと思い
istgt-20111008を使ってzfsのストレージにiSCSIのtargetを作成。
これをWinodws Server 2008R2の物理マシンのinitiatorに読ませてローカルドライブにマッピングする。

[plain]NEKOSABA /usr/local/etc/istgt# /usr/local/etc/rc.d/istgt start
Starting istgt.
istgt version 0.4 (20111008)
normal mode
LU1 HDD UNIT
LU1: LUN0 file=/zpool/tank/istgt-disk1, size=10737418240
LU1: LUN0 20971520 blocks, 512 bytes/block
LU1: LUN0 10.0GB storage for iqn.2007-09.jp.nekolove.istgt:disk1
LU1: LUN0 serial 10000001
LU1: LUN0 read cache enabled, write cache enabled
LU1: LUN0 command queuing enabled, depth 32 [/plain]

DiskをOn-lineへ、初期化してフォーマット。

おー、はやーい。
ほぼGigabitのフルスピードに到達してるのでこれが精一杯
以外なのは4kランダムが早いこと。これは使い物になりそうな予感。
10G Ethernetとか欲しくなるなぁ・・・カードはともかくSwitchが大変なことになるけど。

PRIMERGY TX100 S3ようにオンボードのBIOSからiSCSI Boot出来るようなマシンと組み合わせるとか
仮想マシンのディスクに使うとうまうまーになれる・・・気がする。
何よりあんまり金がかからん。HAのテストとかもいけるで。

で、つい先日、IntalのSSD SSDSA2VP020G2(20G,SLC)を特価3980円でげっとしたので
L2ARCとZILキャッシュに使ってみる。
コストパフォーマンスの都合からパーティション分割で一台を共用するー。

ZILキャッシュはメインメモリの半分しか使われないので、4Gだけ確保ー
残りをL2ARC用に取っておく

[plain]NEKOSABA /# gpart create -s gpt ada5
ada5 created
NEKOSABA /var/db# gpart show ada5
=> 34 39091181 ada5 GPT (18G)
34 39091181 – free – (18G)

NEKOSABA /# gpart add -b 34 -s 4G -t freebsd-zfs ada5
ada5p1 added
NEKOSABA /var/db# gpart show ada5
=> 34 39091181 ada5 GPT (18G)
34 8388608 1 freebsd-zfs (4.0G)
8388642 30702573 – free – (14G)

NEKOSABA /# gpart add -b 8388642 -s 30702573 -t freebsd-zfs ada5
ada5p2 added
NEKOSABA /var/db# gpart show ada5
=> 34 39091181 ada5 GPT (18G)
34 8388608 1 freebsd-zfs (4.0G)
8388642 30702573 2 freebsd-zfs (14G) [/plain]

気になっていたので、一応追加した後remove出来るかどうか試してみた。
昔のバージョンだと一度追加したら外せない(poolを壊すしか無い)という情報を見たので・・・。

[plain]NEKOSABA /# zpool add zpool log ada5p1
NEKOSABA /# zpool status
pool: zfsboot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zfsboot ONLINE 0 0 0
ada0p3 ONLINE 0 0 0

errors: No known data errors

pool: zpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
logs
ada5p1 ONLINE 0 0 0

errors: No known data errors

NEKOSABA /# zpool remove zpool ada5p1
NEKOSABA /# zpool status
pool: zfsboot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zfsboot ONLINE 0 0 0
ada0p3 ONLINE 0 0 0

errors: No known data errors

pool: zpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0

errors: No known data errors
[/plain]

どうやら大丈夫そう。
ZILキャッシュは普通に削除出来るようだ。

[plain]NEKOSABA /zpool# zpool add zpool log ada5p1
NEKOSABA /zpool# zpool add zpool cache ada5p2
NEKOSABA /zpool# zpool status
pool: zfsboot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zfsboot ONLINE 0 0 0
ada0p3 ONLINE 0 0 0

errors: No known data errors

pool: zpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
logs
ada5p1 ONLINE 0 0 0
cache
ada5p2 ONLINE 0 0 0

errors: No known data errors

NEKOSABA /zpool# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
———- —– —– —– —– —– —–
zfsboot 3.80G 141G 1 2 20.6K 19.7K
ada0p3 3.80G 141G 1 2 20.6K 19.7K
———- —– —– —– —– —– —–
zpool 28.1G 7.22T 113 198 14.1M 20.5M
raidz1 28.1G 7.22T 113 198 14.1M 20.5M
ada1 – – 49 97 3.53M 6.84M
ada2 – – 49 97 3.53M 6.84M
ada3 – – 49 97 3.53M 6.84M
ada4 – – 49 97 3.53M 6.83M
ada5p1 0 3.97G 0 0 1.64K 34.0K
cache – – – – – –
ada5p2 352M 14.3G 0 38 1.83K 4.75M
———- —– —– —– —– —– —– [/plain]

ちゃんと動いてるぽい雰囲気。
しかし効果は微妙だった。

キャッシュなしのベンチ

キャッシュありのベンチ

うーむ・・・
やっぱり微妙＾ｑ＾
でもこんなもんかもね、書き込みと4kランダムリードが改善されてるし
分かっていたけど劇的に良くなるようなものではなかった

ある程度環境作ってから計ったもの、参考値
キャッシュ有効、Samba経由

遅い遅いと嘆いていたSambaさんがこんなに早く・・・
以前いろいろ設定変えてどうにか早く出来ないか四苦八苦したというのに。どうしてこうなった＾ｑ＾
忘れないように設定項目を書き出しておこう
影響しそうなのはこの辺かしら。

[plain]NEKOSABA /# uname -a
FreeBSD nekosaba.nekolove.jp 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64 [/plain]

[plain]NEKOSABA /var/log# cat /etc/sysctl.conf
# $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
# This file is read when going to multi-user and its contents piped thru
# “sysctl” to adjust kernel values. “man 5 sysctl.conf” for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0

kern.ipc.somaxconn=4096
kern.maxfiles=65536
kern.maxfilesperproc=32768
net.inet.tcp.rfc1323=1
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
net.inet.udp.recvspace=262144
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=262144
net.local.stream.sendspace=262144[/plain]

[plain]NEKOSABA /# cat /usr/local/etc/smb.conf
# Samba config file created using SWAT
# from UNKNOWN (172.16.11.101)
# Date: 2012/01/24 13:28:40

[global]
dos charset = CP932
display charset = UTF-8
server string = Nekosaba
log file = /var/log/samba/log.%m
max log size = 500
max protocol = SMB2
socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576
max xmit = 1048576
load printers = No
os level = 128
dns proxy = No
idmap config * : backend = tdb

[homes]
comment = Home Directories
valid users = %U
read only = No
create mask = 0644
browseable = No

[share]
comment = ftp directories
path = /zpool/ftp_root
# admin users = nekolove
write list = @ftp_user
writeable = No
create mask = 0664
directory mask = 0775
force group = ftp_user [/plain]

最後にarc_summaryとdmesgでも。

[plain]NEKOSABA /home/bin# ./arc_summary.pl

————————————————————————
ZFS Subsystem Report Tue Jan 24 15:16:27 2012
————————————————————————

System Memory:

7.59% 594.30 MiB Active, 4.15% 324.72 MiB Inact
64.58% 4.94 GiB Wired, 0.07% 5.62 MiB Cache
23.60% 1.80 GiB Free, 0.01% 992.00 KiB Gap

Real Installed: 8.00 GiB
Real Available: 98.86% 7.91 GiB
Real Managed: 96.68% 7.65 GiB

Logical Total: 8.00 GiB
Logical Used: 73.41% 5.87 GiB
Logical Free: 26.59% 2.13 GiB

Kernel Memory: 4.74 GiB
Data: 99.58% 4.72 GiB
Text: 0.42% 20.62 MiB

Kernel Memory Map: 7.16 GiB
Size: 65.09% 4.66 GiB
Free: 34.91% 2.50 GiB
Page: 1
————————————————————————

ARC Summary: (HEALTHY)
Storage pool Version: 28
Filesystem Version: 5
Memory Throttle Count: 0

ARC Misc:
Deleted: 13
Recycle Misses: 0
Mutex Misses: 0
Evict Skips: 0

ARC Size: 70.14% 4.66 GiB
Target Size: (Adaptive) 100.00% 6.65 GiB
Min Size (Hard Limit): 12.50% 850.67 MiB
Max Size (High Water): 8:1 6.65 GiB

ARC Size Breakdown:
Recently Used Cache Size: 73.35% 4.87 GiB
Frequently Used Cache Size: 26.65% 1.77 GiB

ARC Hash Breakdown:
Elements Max: 59.68k
Elements Current: 100.00% 59.68k
Collisions: 43.70k
Chain Max: 6
Chains: 10.36k
Page: 2
————————————————————————

ARC Efficiency: 1.64m
Cache Hit Ratio: 99.22% 1.63m
Cache Miss Ratio: 0.78% 12.79k
Actual Hit Ratio: 98.88% 1.62m

Data Demand Efficiency: 99.71% 1.57m
Data Prefetch Efficiency: 45.88% 1.61k

CACHE HITS BY CACHE LIST:
Anonymously Used: 0.34% 5.61k
Most Recently Used: 13.23% 215.17k
Most Frequently Used: 86.42% 1.41m
Most Recently Used Ghost: 0.00% 0
Most Frequently Used Ghost: 0.00% 0

CACHE HITS BY DATA TYPE:
Demand Data: 96.47% 1.57m
Prefetch Data: 0.05% 741
Demand Metadata: 3.18% 51.73k
Prefetch Metadata: 0.30% 4.87k

CACHE MISSES BY DATA TYPE:
Demand Data: 35.75% 4.57k
Prefetch Data: 6.83% 874
Demand Metadata: 26.95% 3.45k
Prefetch Metadata: 30.47% 3.90k
Page: 3
————————————————————————

L2 ARC Summary: (HEALTHY)
Passed Headroom: 40.40k
Tried Lock Failures: 12.30k
IO In Progress: 0
Low Memory Aborts: 0
Free on Write: 247
Writes While Full: 926
R/W Clashes: 0
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 18.30m

L2 ARC Size: (Adaptive) 2.68 GiB
Header Size: 0.00% 0

L2 ARC Breakdown: 12.45k
Hit Ratio: 0.00% 0
Miss Ratio: 100.00% 12.45k
Feeds: 3.10k

L2 ARC Buffer:
Bytes Scanned: 1.85 TiB
Buffer Iterations: 3.10k
List Iterations: 189.68k
NULL List Iterations: 27

L2 ARC Writes:
Writes Sent: 100.00% 1.01k
Page: 4
————————————————————————

File-Level Prefetch: (HEALTHY)

DMU Efficiency: 3.59m
Hit Ratio: 75.51% 2.71m
Miss Ratio: 24.49% 878.98k

Colinear: 878.98k
Hit Ratio: 0.01% 98
Miss Ratio: 99.99% 878.88k

Stride: 2.60m
Hit Ratio: 100.00% 2.60m
Miss Ratio: 0.00% 3

DMU Misc:
Reclaim: 878.88k
Successes: 0.14% 1.25k
Failures: 99.86% 877.64k

Streams: 114.21k
+Resets: 0.01% 11
-Resets: 99.99% 114.20k
Bogus: 0
Page: 5
————————————————————————

ZFS Tunable (sysctl):
kern.maxusers 384
vm.kmem_size 8209674240
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 6368256
vfs.zfs.mfu_ghost_metadata_lsize 38912
vfs.zfs.mfu_ghost_size 6407168
vfs.zfs.mfu_data_lsize 168340992
vfs.zfs.mfu_metadata_lsize 4911616
vfs.zfs.mfu_size 175333376
vfs.zfs.mru_ghost_data_lsize 1260356608
vfs.zfs.mru_ghost_metadata_lsize 1940992
vfs.zfs.mru_ghost_size 1262297600
vfs.zfs.mru_data_lsize 4706902528
vfs.zfs.mru_metadata_lsize 60765696
vfs.zfs.mru_size 4795987968
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 57856
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 1783983104
vfs.zfs.arc_meta_used 129660696
vfs.zfs.arc_min 891991552
vfs.zfs.arc_max 7135932416
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 25475223552
vfs.zfs.write_limit_max 1061467648
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.mg_alloc_failures 8
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.scrub_limit 10
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
Page: 7
————————————————————————[/plain]

[plain]NEKOSABA /home/bin# dmesg
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012
[email protected]:/usr/obj/usr/src/sys/GENERIC amd64
can’t re-use a leaf (hwpstate_verbose)!
module_register: module cpu/ichss already exists!
Module cpu/ichss failed to register: 17
module_register: module cpu/powernow already exists!
Module cpu/powernow failed to register: 17
module_register: module cpu/est already exists!
Module cpu/est failed to register: 17
module_register: module cpu/hwpstate already exists!
Module cpu/hwpstate failed to register: 17
module_register: module cpu/p4tcc already exists!
Module cpu/p4tcc failed to register: 17
CPU: Intel(R) Core(TM) i5 CPU 650 @ 3.20GHz (3192.04-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x20652 Family = 6 Model = 25 Stepping = 2
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x298e3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,AESNI>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8154890240 (7777 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 5
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <HP ProLiant> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xdfd02000-0xdfd023ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0: <Intel PCH USB 2.0 controller USB-B> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pcib2: failed to allocate initial I/O port window: 0-0xfff
pci16: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
pcib3: failed to allocate initial I/O port window: 0-0xfff
pci26: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci28: <ACPI PCI bus> on pcib4
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 18 at device 0.0 on pci28
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci30: <ACPI PCI bus> on pcib5
bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xdf900000-0xdf90ffff irq 19 at device 0.0 on pci30
bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5784 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 68:b5:99:e4:d0:36
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci32: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x2000-0x201f mem 0xdfa20000-0xdfa3ffff,0xdfa00000-0xdfa1ffff irq 16 at device 0.0 on pci32
em0: Using an MSI interrupt
em0: Ethernet address: 00:15:17:51:c3:a8
em1: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x2020-0x203f mem 0xdfa60000-0xdfa7ffff,0xdfa40000-0xdfa5ffff irq 17 at device 0.1 on pci32
em1: Using an MSI interrupt
em1: Ethernet address: 00:15:17:51:c3:a9
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xdfd03000-0xdfd033ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1: <Intel PCH USB 2.0 controller USB-A> on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci48: <ACPI PCI bus> on pcib7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel 5 Series/3400 Series AHCI SATA controller> port 0x1830-0x1837,0x1824-0x1827,0x1828-0x182f,0x1820-0x1823,0x1800-0x181f mem 0xdfd04000-0xdfd047ff irq 18 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
acpi_button0: <Power Button> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ipmi1: <IPMI System Interface> on isa0
device_attach: ipmi1 attach returned 16
ipmi1: <IPMI System Interface> on isa0
device_attach: ipmi1 attach returned 16
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ipmi0: IPMI device rev. 1, firmware rev. 4.21, version 2.0
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <VB0160EAVEQ HPG0> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <Hitachi HDS5C3020ALA632 ML6OA580> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HDS5C3020ALA632 ML6OA580> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <Hitachi HDS5C3020ALA632 ML6OA5C0> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <Hitachi HDS5C3020ALA632 ML6OA5C0> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <INTEL SSDSA2VP020G2 2CV102M5> ATA-7 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 19087MB (39091248 512 byte sectors: 16H 63S/T 16383C)
ada5: Previously was known as ad14
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 12468918 Hz quality 1000
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <vendor 0x099a> at usbus0
ukbd0: <vendor 0x099a USB Keyboard, class 0/0, rev 1.10/1.00, addr 3> on usbus0
kbd0 at ukbd0
uhid0: <vendor 0x099a USB Keyboard, class 0/0, rev 1.10/1.00, addr 3> on usbus0
ugen1.3: <ServerEngines> at usbus1
ukbd1: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 3> on usbus1
kbd2 at ukbd1
ums0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 3> on usbus1
ums0: 8 buttons and [XYZ] coordinates ID=0
Trying to mount root from zfs:zfsboot []…
em0: link state changed to UP[/plain]