Saturday, February 2, 2013

Friday, November 16, 2012

Live Workflow

I feel like I'm pretty proficient at ad-lib, so setting up a composition consisting of samples to play to has always been something I've wanted to try. I got the Ableton Launchpad a few weeks back and have been piecing together samples (of my own,) to use as an ad-lib composition track. It's nice to have the flexibility, control, and be able to play on top of it. Check it out.

Tuesday, September 11, 2012

ZFS - Remote Snapshot Restore over SSH

This is no new topic by any means, but I thought I'd post about it anyways.  For anyone who is running a mirrored or raidx configuration for local home storage, you might find this interesting.  I run FreeBSD (and have for nearly a decade now,) as my primary home server OS.  One of the many many benefits is the ability to use ZFS as a storage solution. I won't go into detail on ZFS, do a search, you'll learn plenty.

For the last couple of years (to save energy/$) I have been running my FreeBSD server as a virtual machine in Parallels on OSX...  Why would I do such a thing?  Where I lived last, the electric bill was absolutely outrageous and it had little to do with the consumption.  Here in socal, each area is billed based on geographic location.  They implement their own unique version of electric grid socialism this way by charging those near the coast (that rarely run A/C or any other high-draw utilities) approximately 1/3 more per kwh than people living further east.  As you go east, down goes the per kwh pricing.  It's more complicated than that, but you get the gist. It's one of the many "perks" I get for living in California.

So, to eliminate as many 24/7 power supplies and to reduce that bill, I moved from my dedicated server, to my mac pro (both of which run 24/7.)  To do this and not lose the features that I so Love about FreeBSD/ZFS/etc..., I spun up a Parallels virtual machine and imaged (using dd) my physical server to the virtual server.  The tricky part comes in moving from two physical disks to virtual disks on the mac. I have two 1TB drives in a zfs mirror configuration that source nfs on my network to all of our mac/linux nodes. So when i created the partitions in mac, I had to add a virtual disk to each of the 1TB drives.  Anyone familiar with disk/os/filesystem performance is cringing now, but knowing what I'm doing and that I don't need high performance on this network, this was a perfectly acceptable solution.
Bottom line, I migrated my disks to virtual disks np, and made my freebsd virtual machine the internet gateway for the my network (including the parent OSX OS,) with a lot of fun networking/routing. Again too much to get into here..

Then I moved East a bit after buying a home (woot!)  I'm not as interested now in running in this configuration, so I'm moving back to a dedicated server solution for a number of reasons, one of which includes home automation, more on that later.

So now I need to reverse this process.  However, it's not as simple as just pulling the disks from the mac and slapping them into my dedicated server because the disks are formated as typical mac extended journaled format.  So I decided to do the hot foot/cold foot shuffle on my drives.


Pre-Step - Build Latest/Greatest FreeBSD on OS drive on Dedicated server and sync over the relevant bits - Out of scope for this post.

Step 1.  - Source Server - BACKUP EVERYTHING to an external source (preferably multiple)

Step 2.  - Source Server - Cleanly shut down Source Server, pull one of the two mirrored physical disks, and install it in the destination server

Step 3.  - Destination Server - Boot, fdisk the new drive and create the zpool
#fdisk -BI /dev/ada1

#zpool create data /dev/ada1

Step 4. - Source Server - Boot it back up and snapshot the existing zfs partitions.  I have a good number of these, so I wanted to make this quick.
#cd /data/

[root@server /data]# ls -lha
total 2708
drwxrwxrwt   16 nunya    wheel         17B Apr 29 12:36 .
drwxr-xr-x   23 nunya       wheel        512B Sep  3 09:10 ..
drwxr-xr-x    3 nunya  wheel          4B Apr 29 15:02 backup
drwxrw-r--   10 nunya    wheel         12B Apr  1 06:38 bin
drwxrwxr-x   11 nunya    wheel         17B Jun 15 08:28 docs
drwxrwxr-x    5 nunya    wheel          8B Apr 28 13:15 down
drwxr-xr-x    3 nunya    wheel          4B Nov 22  2011 git
drwxrwxr-x    8 nunya    messagebus     8B Aug 29  2011 jails
drwxrwxr-x    8 nunya    wheel         10B Sep 26  2010 movies
drwxrwxrwx  831 nunya    wheel        880B Feb  1  2012 mp3
drwx------    2 nunya    wheel          2B Apr  6  2011 mysql
drwxr-xr-x   15 nunya       wheel         17B May 27 13:46 opt
drwxrwxr-x   13 nunya    wheel         22B May 13 03:00 photos
drwxrwxr-x    2 nunya    wheel          2B Jun 30  2009 projects
drwxr-xr-x    2 nunya       wheel          4B Nov 18  2011 server
drwxrwxr-x    9 nunya    wheel          9B Feb  7  2010 svn
-rwxrwxr-x    1 nunya    wheel        2.5M Jun 28  2009 zfs.core
#for i in `ls`;do zfs snapshot data/$i@1;done

So now I have snapshots of each zfs partition. Now for the beauty of ZFS's magic... I'm ready to start transferring data from my virtual disk on my mac, to the real mccoy on my destination server.

5. Source Server - Sync those snapshots directly over ssh to the destination server
#for i in `ls |grep -v zfs.core`;do zfs send data/$i@1 |ssh server2 zfs receive data/$i ;done

Voila, checking the zfs list on the destination server, I can see the partitions being created and populated with block level data... A perfect copy....  Go get lunch, at nearly 500GB of data, this is going to take awhile, even on GB network.


After a refreshing lunch, and a short nap, I've got a physical disk with the replicated ZFS Volumes on the destination machine. Now I need to get the other physical disk into the new machine. I shutdown both machines, moved the physical drive over, and repeated the fdisk command above.
#fdisk -BI /dev/ada2


Now I need to add the new disk to the zpool
#zpool attach data ada2 ada1


I ran a few zpool status checks and watched the first disk replicate to the second disk. After a couple of hours:
server# zpool status
  pool: data
 state: ONLINE
 scan: resilvered 422G in 1h32m with 0 errors on Tue Sep  4 19:27:23 2012
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada1    ONLINE       0     0     0

errors: No known data errors



And there we have it kids. Now I have migrated a zfs raid pool from one server to the other. Just need to set up the sharing again
#zfs set sharenfs=on data/bin
#zfs set sharenfs=on data/docs
#zfs set sharenfs=on data/down
#zfs set sharenfs=on data/movies
#zfs set sharenfs=on data/mp3
#zfs set sharenfs=on data/photos
#zfs set sharenfs=on data/projects

And we are done. Enjoy.

Monday, May 14, 2012

Picasa Duplicates and fdupes

I'm on Snow Leopard and was having troubles with picasa importing an nfs directory.  Apparently in the last upgrade of picasa, it hosed my watched directories.  

I found How to manually add directories to Picasa's watch list and it worked great... too great...  As soon as I opened it up I realized that this directory had been partially imported by picasa before and I started seeing a massive number of duplicate photos.  

With any duplicate removal, there is risk that two different pictures could share a name or two names could share the same content in one or more of the directories and finding the duplicate by name alone is not a solution for identifying duplicates.  Enter fdupes. Fdupes analyzes the content of a file using an md5 hash and compares the hash between two files. If they match, they are truly a duplicate and can be dealt with accordingly. I did a bit of research on the best method of using fdupes because if you just run it as is, it will look for user interaction throughout this process. With large datasets, this isn't feasible. What I landed on came from Here.

If you're on mac and have macports installed (highly recommended if you're comfortable at the command line,) you can:
  • Install fdupes

#port -dv install fdupes
  • Backup Your Picasa Content Directory. For most this just means making sure Time Machine is up to date. For me this meant:
#rsnapshot -v hourly
  • Now run fdupes against your picasa directory deleting indiscriminately.. (make sure you have a backup)

#yes 2 | fdupes -dr .
Having more than 10k files to check (many of them video,) my job took a WHILE