Blogs

 

Storage: Ephemeral or EBS?

When you launch an RHEL instance on EC2, the disk storage that is provided is a VERY SMALL (7 GB or less) root partition that is contained within the instance itself. This type of storage is called ephemeral. The benefit of ephemeral storage is that it doesn't cost any extra money. The drawback is that it can't participate in the EBS backup solution provided by Amazon -- you'll need to backup files across the network instead. Depending on the size of the instance, you will be given additional ephemeral storage, which is where you can potentially store data or install an app server for Liferay. The Amazon documentation indicates that this will be mounted automatically on /ephemeral0 but I found that this wasn't the case. You will need to use the "fdisk -l" command to find out the device name of the additional ephemeral storage. In my case, I found that the device was named /dev/xvdj. However, there was no filesystem on it, and it wasn't mounted. So you'll need to execute commands like the following:

mkfs -t ext4 /dev/xvdj

mkdir /ephemeral0

echo "/dev/xvdj /ephemeral0 ext4 defaults 1 2" >> /etc/fstab

mount /ephemeral0

 

The alternative storage is called EBS. The benefit is that it can participate in the Amazon backup solution, but the drawback is that it costs extra money. If you want to use EBS instead of ephemeral, you can create EBS Volumes using the Amazon Management Console and then attach them to your instance. Bear in mind that you'll still need to discover the device with "fdisk -l", put a filesystem on it, mount it, etc.

 

Moving MySQL to the additional ephemeral storage

In order to utilize the larger ephemeral storage for database files, I needed to move the MySQL data files from /var/lib/mysql to a filesystem mounted on /ephemeral0. I tried to follow the instructions in section 10.4.1 of the HREL documentation titled " MySQL Changing Database Location" but when I restarted mysqld via "service mysqld start" it reported FAILED, and the following was found in /var/log/mysqld.log:

 

110902 11:35:42 mysqld_safe Starting mysqld daemon with databases from /opt/var/mysql

110902 11:35:42 [Warning] Can't create test file /opt/var/mysql/domU-12-31-39-09-AE-12.lower-test

110902 11:35:42 [Warning] Can't create test file /opt/var/mysql/domU-12-31-39-09-AE-12.lower-test

/usr/libexec/mysqld: Can't change dir to '/opt/var/mysql/' (Errcode: 13)

110902 11:35:42 [ERROR] Aborting

 

The irony here is that section 10.4.1 is meant to show you how to avoid this error.  So I had to find a different approach, and found inspiration in the following articles:

 

The strategy I used was to copy the MySQL files to /ephemeral0, and then mount the ephemeral storage directory back to /var/lib/mysql

 

Step 1: Stop the MySQL server and copy the data files over to /ephemeral0:

[root@server]# service mysqld stop

[root@server]# cd /var/lib

[root@server]# tar cf - mysql/ --selinux | (cd /ephemeral0/var/lib; tar xvfp -)

NOTE: It's really important to keep the --selinux switch so that the SELinux context for each file will be saved in the tarball. Also, it's really important to keep the "p" switch upon extraction, so that file permissions will be preserved.

 

Step 2: Determine the existing selinux settings and re-apply them on a similar var/lib folder structure on /ephemeral0:

[root@server]# ls -lZ / | grep var

drwxr-xr-x. root root system_u:object_r:var_t:s0       var

 

[root@server]# chcon -u system_u -r object_r -t var_t /ephemeral0/var

 

[root@server]# ls -lZ /var | grep lib

drwxr-xr-x. root   root   system_u:object_r:var_lib_t:s0   lib

 

[root@server]# chcon -u system_u -r object_r -t var_t /ephemeral0/var/lib

 

[root@server]# ls -lZ /var/lib | grep mysql

drwxr-xr-x. mysql     mysql    system_u:object_r:mysqld_db_t:s0 mysql

 

Step 3: Re-create the /var/lib/mysql mount point and preserve ownership/group/selinux settings:

[root@server]# cd /var/lib

[root@server]# mv mysql mysql.orig

[root@server]# mkdir mysql

[root@server]# chown mysql mysql

[root@server]# chgrp mysql mysql

[root@server]# chcon -u system_u -r object_r -t mysqld_db_t mysql

 

Step 4: Mount the directory that contains the MySQL data onto the /var/lib/mysql mount point and restart MySQL:

[root@server]# echo "/ephemeral0/var/lib/mysql /var/lib/mysql none bind bind" >> /etc/fstab

[root@server]# mount /var/lib/mysql

[root@server]# service mysqld start