With all of this tutorial publishing I’ve been doing for Grafana, I have neglected to post what mine currently looks like.  So, here it is.  This is the 1080P version.  I also built an iPad version that has a lot of the same info but compressed for the smaller screen.

Continuing the documentation effort.  This is a shell script you run from Unraid in a cron job to feed stats to InfluxDB.  You can then present them in Grafana.  Note about that, I was having a lot of trouble getting the Grafana graphs to present correctly for anything coming from this script.  I had to change the Fill from “null” to “none” in the graph.  Not sure why that’s happening, but “none” gets it to behave just like everything else.

## Assembled from this post: https://lime-technology.com/forum/index.php?topic=52220.msg512346#msg512346

## add to cron like:

## * * * * * sleep 10; /boot/custom/influxdb.sh > /dev/null 2>&1

## //0,10 * * * * /boot/custom/influxdb.sh > /dev/null 2>&1
#

# Set Vars

#

DBURL=http://192.168.x.x:8086 ## IP address of your InfluxDB server

DBNAME=dashboard ## Easier if you pick an existing DB

DEVICE=”UNRAID”

CURDATE=`date +%s`

# Current array assignment.

# I could pull the automatically from /var/local/emhttp/disks.ini

# Parsing it wouldnt be that easy though.

DISK_ARRAY=( sdn sdl sdf sdc sdj sde sdo sdh sdi sdd sdk sdm sdg sdp sdb )

DESCRIPTION=( parity disk1 disk2 disk3 disk4 disk5 disk6 disk7 disk8 disk9 disk10 disk11 disk12 disk13 cache )

#

# Added -n standby to the check so smartctl is not spinning up my drives

#

i=0

for DISK in “${DISK_ARRAY[@]}”

do

smartctl -n standby -A /dev/$DISK | grep “Temperature_Celsius” | awk ‘{print $10}’ | while read TEMP

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “DiskTempStats,DEVICE=${DEVICE},DISK=${DESCRIPTION[$i]} Temperature=${TEMP} ${CURDATE}000000000” >/dev/null 2>&1

done

((i++))

done
# Had to increase to 10 samples because I was getting a spike each time I read it. This seems to smooth it out more

top -b -n 10 -d.2 | grep “Cpu” | tail -n 1 | awk ‘{print $2,$4,$6,$8,$10,$12,$14,$16}’ | while read CPUusr CPUsys CPUnic CPUidle CPUio CPUirq CPUsirq CPUst

do

top -bn1 | head -3 | awk ‘/load average/ {print $12,$13,$14}’ | sed ‘s/,//g’ | while read LAVG1 LAVG5 LAVG15

do

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “cpuStats,Device=${DEVICE} CPUusr=${CPUusr},CPUsys=${CPUsys},CPUnic=${CPUnic},CPUidle=${CPUidle},CPUio=${CPUio},CPUirq=${CPUirq},

CPUsirq=${CPUsirq},CPUst=${CPUst},CPULoadAvg1m=${LAVG1},CPULoadAvg5m=${LAVG5},CPULoadAvg15m=${LAVG15} ${CURDATE}000000000” >/dev/null 2>&1

done

done
if [[ -f byteCount.tmp ]] ; then
# Read the last values from the tmpfile – Line “eth0”

grep “eth0” byteCount.tmp | while read dev lastBytesIn lastBytesOut

do

cat /proc/net/dev | grep “eth0” | grep -v “veth” | awk ‘{print $2, $10}’ | while read currentBytesIn currentBytesOut

do

# Write out the current stats to the temp file for the next read

echo “eth0” ${currentBytesIn} ${currentBytesOut} > byteCount.tmp
totalBytesIn=`expr ${currentBytesIn} – ${lastBytesIn}`

totalBytesOut=`expr ${currentBytesOut} – ${lastBytesOut}`
curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “interfaceStats,Interface=eth0,Device=${DEVICE} bytesIn=${totalBytesIn},bytesOut=${totalBytesOut} ${CURDATE}000000000” >/

dev/null 2>&1
done

done
else

# Write out blank file

echo “eth0 0 0” > byteCount.tmp

fi
# Gets the stats for boot, disk#, cache, user

#

df | grep “mnt/\|/boot\|docker” | grep -v “user0\|containers” | sed ‘s/\/mnt\///g’ | sed ‘s/%//g’ | sed ‘s/\/var\/lib\///g’| sed ‘s/\///g’ | while read MOUNT TOTAL USED FREE UTILIZATION DISK

do

if [ “${DISK}” = “user” ]; then

DISK=”array_total”

fi

curl -is -XPOST “$DBURL/write?db=$DBNAME” –data-binary “drive_spaceStats,Device=${DEVICE},Drive=${DISK} Free=${FREE},Used=${USED},Utilization=${UTILIZATION} ${CURDATE}000000000” >/dev/null 2>&

1

done

Following my previous post about Grafana, once everything is installed you’ll want to capture some data.  Otherwise, what’s the point.  Telegraf is a data gathering tool made by Influxdata.  It’s stupid simple to get working with InfluxDB.  After following the previous script, go to /etc/telegraf/ and edit telegraf.conf.  Near the top is the Output Plugins section.  Make sure that’s modified for your InfluxDB install.  From there, scroll down to Input Plugins.  There’s a ridiculous number of input plugins available.  We’re focused on SNMP today, but it’s worth looking through the list to see if a “need” can be solved with Telegraf before using some other custom script.

For me, I needed to add SNMP for my Ubiquiti ER-X firewall and my Nutanix CE cluster.  Here’s my SNMP config section with the obvious security bits redacted:

# # Retrieves SNMP values from remote agents
# [[inputs.snmp]]
[[inputs.snmp]]
agents = [ “192.168.x.x:161” ] ##Nutanix CE CVM IP
timeout = “5s”
version = 3

max_repetitions = 50

sec_name = “username”
auth_protocol = “SHA” # Values: “MD5”, “SHA”, “”
auth_password = “password”
sec_level = “authPriv” # Values: “noAuthNoPriv”, “authNoPriv”, “authPriv”

priv_protocol = “AES” # Values: “DES”, “AES”, “”
priv_password = “password”

name = “nutanix”
[[inputs.snmp.field]]
name = “host1CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.1”
[[inputs.snmp.field]]
name = “host2CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.2”
[[inputs.snmp.field]]
name = “host3CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.3”
[[inputs.snmp.field]]
name = “host4CPU”
oid = “1.3.6.1.4.1.41263.9.1.6.4”
[[inputs.snmp.field]]
name = “ClusterIOPS”
oid = “1.3.6.1.4.1.41263.506.0”
[[inputs.snmp.field]]
name = “Host1MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.1”
[[inputs.snmp.field]]
name = “Host2MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.2”
[[inputs.snmp.field]]
name = “Host3MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.3”
[[inputs.snmp.field]]
name = “Host4MEM”
oid = “1.3.6.1.4.1.41263.9.1.8.4”

[[inputs.snmp]]
agents = [ “192.168.0.1:161” ] ##Firewall IP
timeout = “5s”
retries = 3
version = 2
community = “RO_community_string”
max_repetitions = 10

name = “ERX”
[[inputs.snmp.field]]

name = “Bytes.Out”
oid = “1.3.6.1.2.1.2.2.1.10.2”
[[inputs.snmp.field]]
name = “Bytes.In”
oid = “1.3.6.1.2.1.2.2.1.16.2”

You’ll have to get Telegraf to read in the config again.  The sledgehammer method would be a reboot.  I think a Telegraf service restart would also do the trick.  Reboots for me take about 5 seconds (yep, really), so it’s useful to make sure it’s coming up clean on a reboot anyway.

Just went through setting up Grafana on Ubuntu 16.04 and thought I would grab the steps I went through.  I’m using a combination of Telegraf and some custom remote scripts to get data into InfluxDB.

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add –
source /etc/lsb-release
echo “deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable” | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install influxdb
sudo service influxdb start
echo “deb https://packagecloud.io/grafana/testing/debian/ wheezy main” | sudo tee /etc/apt/sources.list.d/grafana.list
curl https://packagecloud.io/gpg.key | sudo apt-key add –
sudo apt-get update && sudo apt-get install grafana
sudo service grafana-server start
wget https://dl.influxdata.com/telegraf/releases/telegraf_1.2.1_amd64.deb
sudo dpkg -i telegraf_1.2.1_amd64.deb
telegraf -sample-config > telegraf.conf
nano telegraf.conf
telegraf -config telegraf.conf
sudo cp telegraf.conf /etc/telegraf/telegraf.conf
sudo systemctl enable grafana-server.service
sudo systemctl enable telegraf.service
sudo reboot

This gets things installed.  I’ll have another post to describe other configuration that’s required.

I’ve been running on a Nutanix CE install for about a month now.  With the November release they added some much needed GUI controls for the image service.  You can now import ISOs for install images, without having to fiddle with CLI stuff.

I’ve had virtually no problems, and the VMs are performing well.  If there’s one complaint I have with this solution it’s that the baseline memory utilization is high.  I couldn’t reduce the CVM’s to less than 8GB each without running into serious problems with the cluster.  Plus, there seems to be a missing 3GB per host.  I’m assuming this is what the actual CE and KVM host requires, but that seems high.  I know I can run VMWare ESXi in less than 1GB per host.  So, 11GB per host is used up right from the start.  Since I’m running this on a shoestring budget with 16GB per host, I really only have 5GB available for VMs.  That kinda sucks.

On the upside, the CVM’s at 8GB work fine and the IO performance is pretty amazing.  I’ve seen upwards of 1600 IOPS at times.  This is basically a single consumer grade 240GB SSD in each host for the primary tier and 640GB HDD for the secondary tier.  I don’t think I’m even using the secondary yet.  3 hosts at varying levels of i5 CPU’s, but none of them current gen.

I’m pretty happy with this and I’m looking forward to seeing what Nutanix does next.

The Nutanix install has been moving along.  I would not say it’s ready for more than lab use, but it’s getting there.  I’m setting up a 3 node cluster, and one of the nodes, which has an Intel motherboard, kept throwing a generic error about not being able to find the sysinfo.  Thanks to the help from the forum, I was able to hard code a product name in order to get past the install.  I don’t think it will have an impact on operation, only install, but it’s one of those little things that crops up with new software.

The link is here, if you’re able to access it:  http://next.nutanix.com/t5/Discussion-Forum/Install-failing-with-quot-unable-to-gather-basic-hardware/td-p/5034/page/2

 

About 2 months ago Nutanix.com released a free software only version of their magic, called Community Edition.  I got on the list for this as quickly as I could, but I haven’t been able to install it until now.  See, I wanted to have an actually cluster, what the call RF2 (Redundancy Factor), which would require me to blow away my existing XenServer install to get to enough compatible hardware.  I also needed to purchase SSD’s for each of the nodes in the cluster.

Well, I’ve done that now.  At the moment, I’m exporting my VM’s out of XenServer to OVA’s, in the hope I can restore them from that.  If I can’t, well….I’m not sure then.  I may just rebuild everything from scratch.  I’d really like to figure out how to import them, though.

What I’ll have when I’m done is a 3 node RF2 cluster, with the minimum a 240GB SSD, and at least a 500GB HDD in each node.   All 3 nodes are i5’s, of different vintages. Not a lot of space, once you run the Nutanix overhead, but it’ll be enough for my needs.  I’ll post some screenshots and pics once I’m up and running.

I’ve had my Plex Media Server running on Ubuntu 12.04 on XS 6.2 for a while now.  It recently started nagging me to upgrade to 14.04.  Didn’t see a reason not to do this, but I also didn’t do much digging to see if there were known issues.  Turns out, 14.04 is not a “supported” version on XS 6.2.  The updated image refused to boot.  It ended up being a pretty simple fix of going into the XS host and modifying a Grub configuration file.  It was an easy thing to find through googling.  First item in the results.

I should also mention, the Ubuntu Dist Upgrade also uninstalled Plex.  That was an easy fix, but it did require a reboot for Plex to start working.

I actually did this a little while ago and forgot to post about it.  I could not get NFS to work between XenServer and Unraid.  So, I scrounged up some hardware and punted.  I installed OpenFiler and iSCSI.

And it worked.

Angels sang.  Rainbows shone.  Something about unicorns…

It’s been working great.  I just have a new problem.  The hardware I’ve been using for XenServer is my old PC hardware.  It’s 1st gen i5 so it should be plenty for my limited requirements, and it is.  It’s just getting hung randomly now.  I had this problem when it was my PC.  In fact, that’s why I upgraded.  Looks like it’s hardware related.  So, now I’m contemplating getting a new MoBo and CPU.  What a drag.

The good news is, I’m pretty sure I’ll be able to go to all XenServer and get rid of the VMWare, all while retaining live migration across the iSCSI pool.  VMWare, I’d love to stick with you, but I need live migration in the free version.

Being on a XenServer has another benefit.  I might try something like Puppet, just for kicks.  🙂

Just…can’t…get…it…to…work.  So, I’m going to try XenServer free again and ask for help at the support forums for both Citrix and Unraid.  I managed to get noac to populate into the NFS settings but lookupcache=none isn’t taking.  There are other settings in there and I’m wondering if they’re getting in the way.  I have to believe there’s a way to make this work.  From the nfsstat -m on the server:

/var/run/sr-mount/1804503a-ac43-f0eb-cd4a-18eff5098f21 from 192.168.169.8:/mnt/user/ISOs
Flags: rw,sync,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,soft,noac,proto=tcp,port=65535,timeo=133,retrans=0,sec=sys,mountport=65535,local_lock=none,addr=192.168.169.8