OpenIO : Add a New Volume to Existing Infrastructure




Background

You are archiving (a lot), and soon, your archiving platform become full. How to solve this ? Following the below procedure, you will add an extra volume to the OpenIO platform and continue to archive, again and again.

Let's add some storage

First, you need to add the storage on the node itself. This can be done with physical disks or adding a virtual one with your hypervisor. I'm supposing this step is done and the new device is already seen by the OS.

Step 1 - Create a new disk and add XFS file system on it

[root@be-openio-01 ~]# fdisk -l

[...]
Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes


In our case, this is /dev/sdc, let's create a new partition with fdisk and add file system with mkfs.xfs

[root@be-openio-01 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x3a440cd0.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199
Partition 1 of type Linux and of size 100 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.


Syncing disks.


Let's add a file system on it


[root@be-openio-01 ~]# mkfs.xfs /dev/sdc1
meta-data=/dev/sdc1              isize=512    agcount=4, agsize=6553536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=26214144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=12799, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

mount new volume

Next, we need to create an additional mount point and mount the volume

[root@be-openio-01 ~]# mkdir /data2
[root@be-openio-01 ~]# mount /dev/sdc1 /data2
[root@be-openio-01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
[...]
/dev/sdc1                100G   33M  100G   1% /data2

We also need to add it to /etc/fstab so the volume will automatically be mounted at boot time.

/dev/sdc1       /data2 xfs defaults 1 1

Last step in the storage preparations tasks is creating the relevant folders and set permissions on them.

[root@be-openio-01 ~]# mkdir /data2/sds
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO/rdir-2
[root@be-openio-01 ~]# mkdir /data2/sds/OPENIO/rawx-2
[root@be-openio-01 ~]# chown -R openio:openio /data2

Now, we have the new storage ready on our server. We can start telling OpenIO to make good use of it !

Step 2 - Modify the puppet manifest with new disk details

Open the openio.pp manifest you used for the initial deployment and :
  • Change the service name : rawx-0 -> rawx-1, ....
  • Increment the "num"
  • Adjust the documentRoot path with the new mount point you have created above
openiosds::rawx {'rawx-2':
  num => 2,
  port => 6203,
  ns => 'OPENIO',
  ipaddress => $ipaddr,
  documentRoot => '/data2/sds/OPENIO/rawx-2',
}
openiosds::rdir{'rdir-2':
  num => 2,
  port => 6303,
  ns => 'OPENIO',
  ipaddress => $ipaddr,
  db_path=> '/data2/sds/OPENIO/rdir-2',
}
openiosds::oioblobindexer{'oio-blob-indexer-rawx-2':
  num => 2,
  ns => 'OPENIO',
  volume=> '/data2/sds/OPENIO/rawx-2',
}

Re-apply puppet manifest

[root@be-openio-01 ~]# puppet apply --no-stringify_facts ~/openio.pp
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for be-openio-01 in environment production in 2.92 seconds
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
[...]
   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
Notice: /Stage[main]/Main/Openiosds::Rdir[rdir-2]/Openiosds::Service[OPENIO-rdir-3]/File[/data2/sds/OPENIO/rdir-2]/owner: owner changed 'root' to 'openio'
Notice: /Stage[main]/Main/Openiosds::Rdir[rdir-2]/Openiosds::Service[OPENIO-rdir-3]/File[/data2/sds/OPENIO/rdir-2]/group: group changed 'root' to 'openio'
[...]
Notice: Finished catalog run in 3.53 seconds

Next, we need to initialise the newly created data volume with the bootstrap mechanism

[root@be-openio-01 ~]# openio volume admin bootstrap --oio-ns OPENIO
+--------------------+--------------------+---------------+---------------+
| Rdir               | Rawx               | Rdir location | Rawx location |
+--------------------+--------------------+---------------+---------------+
| 192.168.8.109:6010 | 192.168.8.110:6004 | be-openio-02  | be-openio-03  |
| 192.168.8.109:6010 | 192.168.8.110:6202 | be-openio-02  | be-openio-03  |
| 192.168.8.109:6010 | 192.168.8.139:6004 | be-openio-02  | be-openio-01  |
| 192.168.8.110:6010 | 192.168.8.109:6004 | be-openio-03  | be-openio-02  |
| 192.168.8.110:6010 | 192.168.8.139:6202 | be-openio-03  | be-openio-01  |
| 192.168.8.139:6302 | 192.168.8.109:6202 | be-openio-01  | be-openio-02  |
+--------------------+--------------------+---------------+---------------+

Lastly, restart the services

[root@be-openio-01 ~]# gridinit_cmd restart
DONE            OPENIO-oio-blob-indexer-3       Success
DONE            OPENIO-rawx-3   Success
DONE            OPENIO-rdir-3   Success
DONE            OPENIO-rawx-2   Success
DONE            OPENIO-rdir-2   Success
DONE            OPENIO-oio-blob-indexer-2       Success
DONE            OPENIO-memcached-0      Success
DONE            OPENIO-oioswift-0       Success
DONE            OPENIO-rawx-0   Success
DONE            OPENIO-conscience-0     Success
DONE            OPENIO-zookeeper-0      Success
DONE            OPENIO-oioproxy-0       Success
DONE            OPENIO-redissentinel-0  Success
DONE            OPENIO-rdir-0   Success
DONE            OPENIO-oio-blob-indexer-0       Success
DONE            OPENIO-oio-event-agent-0        Success
DONE            OPENIO-redis-0  Success
DONE            OPENIO-meta2-0  Success
DONE            OPENIO-meta1-0  Success
DONE            OPENIO-account-0        Success
DONE            OPENIO-conscienceagent-0        Success
DONE            OPENIO-meta0-0  Success
DONE            OPENIO-beanstalkd-0     Success

Never forget to unlock the repositories

[root@be-openio-01 ~]# openio cluster unlockall --oio-ns OPENIO
+---------+--------------------+----------+
| Type    | Service            | Result   |
+---------+--------------------+----------+
| rdir    | 192.168.8.109:6302 | unlocked |
| rdir    | 192.168.8.109:6010 | unlocked |
| rdir    | 192.168.8.110:6010 | unlocked |
| rdir    | 192.168.8.110:6302 | unlocked |
| rdir    | 192.168.8.139:6302 | unlocked |
| rdir    | 192.168.8.139:6010 | unlocked |
| account | 192.168.8.109:6009 | unlocked |
| account | 192.168.8.139:6009 | unlocked |
| account | 192.168.8.110:6009 | unlocked |
| rawx    | 192.168.8.109:6004 | unlocked |
| rawx    | 192.168.8.109:6202 | unlocked |
| rawx    | 192.168.8.110:6004 | unlocked |
| rawx    | 192.168.8.110:6202 | unlocked |
| rawx    | 192.168.8.139:6004 | unlocked |
| rawx    | 192.168.8.139:6202 | unlocked |
| meta2   | 192.168.8.109:6003 | unlocked |
| meta2   | 192.168.8.110:6003 | unlocked |
| meta2   | 192.168.8.139:6003 | unlocked |
| meta1   | 192.168.8.109:6002 | unlocked |
| meta1   | 192.168.8.110:6002 | unlocked |
| meta1   | 192.168.8.139:6002 | unlocked |
| meta0   | 192.168.8.109:6001 | unlocked |
| meta0   | 192.168.8.110:6001 | unlocked |
| meta0   | 192.168.8.139:6001 | unlocked |
+---------+--------------------+----------+

Check if new rawx is coming up (score should increase up to 100 by 5% increments)

[root@be-openio-01 ~]# openio --oio-ns OPENIO cluster list | grep rawx
| rawx    | 192.168.8.139:6203 | /data2/sds/OPENIO/rawx-2        | be-openio-01 | n/a   | True  |    99 |
| rawx    | 192.168.8.139:6202 | /data/sds/OPENIO/rawx-1         | be-openio-01 | n/a   | True  |    88 |
| rawx    | 192.168.8.139:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-01 | n/a   | True  |    86 |
| rawx    | 192.168.8.110:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-03 | n/a   | True  |    86 |
| rawx    | 192.168.8.110:6202 | /data/sds/OPENIO/rawx-1         | be-openio-03 | n/a   | True  |    89 |
| rawx    | 192.168.8.109:6202 | /data/sds/OPENIO/rawx-1         | be-openio-02 | n/a   | True  |    89 |
| rawx    | 192.168.8.109:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | be-openio-02 | n/a   | True  |    82 |

Our new data repository is ready to be used !

Ideally, you need to redo the above procedure on each hosts in your OpenIO cluster to be sure data is balanced accordingly.


Comments

What's hot ?

Raspberry Pi : WiFi Hotspot for the Garden!

ShredOS : HDD degaussing with style

Wallbox : Get The Most Of It (with API)