Gluster active/passive cluster

Gluster is a nice distributed file system which offers some management benefits over block level storage systems like DRBD. By design Gluster works in an active/active cluster configuration, however for applications where millisecond precision data replication is essential, an active/passive configuration is preferable. This is how (based on SLES11 with HAE):

usual SLES HA cluster creation:
create cluster key: corosync-keygen
add nodes to corosync.conf
ensure all nodes are in hosts file
chkconfig openais on

Destroy gluster brick

If you wish to destroy or recreate a Gluster brick, it leaves some traces behind on the filesystem. This is the procedure to blat it:

umount
gluster volume stop gv0
gluster volume delete gv0
attr -lq /data/gv0/brick1
setfattr -x trusted.glusterfs.volume-id /data/gv0/brick1/
setfattr -x trusted.gfid /data/gv0/brick1/
rm –rf /data/gv0/brick1/.glusterfs
/etc/init.d/glusterd restart

Server 2012 R2 – Failover Cluster error

So I was installing SQL2008R2 (In 2015? I know not my choice, it was for a legacy application) into a Windows Server 2012 R2 Failover Cluster. I hit the following error during the SQL install: “Rule Cluster Service Verification Failed. The SQL Server failover cluster services is not online”.

It turns out that some cluster installers (including SQL server) depend on the deprecated cluster.exe which is not installed by default. Additionally you cant add it via the GUI once the Failover Cluster feature has been installed so you need to run the following Powershell cmd to add the subfeatures:

Install-WindowsFeature rsat-clustering-cmdinterface,rsat-clustering-automationserver

Re-run the SQL installer and you should be good to go!

Restart ESXi management agent

ESXi 5.x
/etc/init.d/hostd restart
/etc/init.d/vpxa restart

ESXi 4.x
/etc/init.d/hostd restart
service vmware-vpxa restart

ESX:
service mgmt-vmware restart
service vmware-vpxa restart

Linux open files limit

System-wide file descriptors limits:

View:
cat /proc/sys/fs/file-max

Set:
sysctl -w fs.file-max=100000

Set persistently:
vi /etc/sysctl.conf
fs.file-max = 100000
then run sysctl -p

User level file descriptors limits:

View:
su - httpd
$ ulimit -Hn
$ ulimit -Sn

Set:
vi /etc/security/limits.conf
httpd soft nofile 4096
httpd hard nofile 10240

Force dismount VMware tools

First post in two years, been incredibly busy and blogging always sinks to the bottom of the list!

If VMware tools installer has been mounted to a VM for too long you may experience a scenario where you are unable to unmount it no matter how many times you click end or run dismount-tools. To force the dismount, run the following PowerCLI command. You may need to power off the VM if it doesn’t play ball:

Get-VM | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$false

Load Balanced Linux Web Cluster

Another example of a load balanced apache cluster, this time active-active using Pacemaker/Corosync/ldirectord.
This example is using CentOS.

10.11.1.40 – fixed IP of server1
10.11.1.41 – fixed IP of server 2
10.11.1.80 – apache site 1 (virtual IP)
10.11.1.50 – apache site 1 server 1
10.11.1.51 – apache site 1 server 2
10.11.1.90 – apache site 2 (virtual IP)
10.11.1.60 – apache site 2 server 1
10.11.1.61 – apache site 2 server 2

Some Munin Notes

Munin – its pretty handy for monitoring stuff over time to view trends. Some notes to get it up and running with a minimal config:

Install munin on monitoring server:
apt-get install munin

vi /etc/munin/munin.conf
Add something like:
[Webservers;webserver-01]
address 2.11.1.40
use_node_name yes
[Webservers;webserver-01]
address 2.11.1.40
use_node_name yes

Group by something sensible, i.e server type, location, domain?

Some LVM Notes

A few LVM notes to jog the brain

A. Steps to create the logical volume..

List the current physical volumes. In this instance, were interested in a blank 20GB SATA – /dev/sdb. You can extend a logical volume by adding additional phyical volumes later.

fdisk -l
Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b5a0f

High Availability Linux Web Server Example

Just a quick example of setting up a Linux HA failover environment for an Apache/MySQL web server. This runs through a Ubuntu installation, however RedHat shouldn’t vary too much.

2.11.1.24 – fixed IP of server1
2.11.1.25 – fixed IP of server2
2.11.1.30 – apache site1 (virtual IP)
2.11.1.31 – apache site2 (virtual IP)

Make sure both servers have a sensible hostname and ensure their hosts files relate to this:

etc/hosts
127.0.0.1 localhost
2.11.1.24 ubuntu-server-1.mydomain.com ubuntu-server-1
2.11.1.25 ubuntu-server-2.mydomain.com ubuntu-server-2