Sunday, June 9, 2013
Wednesday, May 29, 2013
Sunday, April 14, 2013
Saturday, February 2, 2013
Friday, November 16, 2012
Live Workflow
I feel like I'm pretty proficient at ad-lib, so setting up a composition consisting of samples to play to has always been something I've wanted to try. I got the Ableton Launchpad a few weeks back and have been piecing together samples (of my own,) to use as an ad-lib composition track. It's nice to have the flexibility, control, and be able to play on top of it. Check it out.
Tuesday, September 11, 2012
ZFS - Remote Snapshot Restore over SSH
This is no new topic by any means, but I thought I'd post about it anyways. For anyone who is running a mirrored or raidx configuration for local home storage, you might find this interesting. I run FreeBSD (and have for nearly a decade now,) as my primary home server OS. One of the many many benefits is the ability to use ZFS as a storage solution. I won't go into detail on ZFS, do a search, you'll learn plenty.
For the last couple of years (to save energy/$) I have been running my FreeBSD server as a virtual machine in Parallels on OSX... Why would I do such a thing? Where I lived last, the electric bill was absolutely outrageous and it had little to do with the consumption. Here in socal, each area is billed based on geographic location. They implement their own unique version of electric grid socialism this way by charging those near the coast (that rarely run A/C or any other high-draw utilities) approximately 1/3 more per kwh than people living further east. As you go east, down goes the per kwh pricing. It's more complicated than that, but you get the gist. It's one of the many "perks" I get for living in California.
So, to eliminate as many 24/7 power supplies and to reduce that bill, I moved from my dedicated server, to my mac pro (both of which run 24/7.) To do this and not lose the features that I so Love about FreeBSD/ZFS/etc..., I spun up a Parallels virtual machine and imaged (using dd) my physical server to the virtual server. The tricky part comes in moving from two physical disks to virtual disks on the mac. I have two 1TB drives in a zfs mirror configuration that source nfs on my network to all of our mac/linux nodes. So when i created the partitions in mac, I had to add a virtual disk to each of the 1TB drives. Anyone familiar with disk/os/filesystem performance is cringing now, but knowing what I'm doing and that I don't need high performance on this network, this was a perfectly acceptable solution.
Bottom line, I migrated my disks to virtual disks np, and made my freebsd virtual machine the internet gateway for the my network (including the parent OSX OS,) with a lot of fun networking/routing. Again too much to get into here..
Then I moved East a bit after buying a home (woot!) I'm not as interested now in running in this configuration, so I'm moving back to a dedicated server solution for a number of reasons, one of which includes home automation, more on that later.
So now I need to reverse this process. However, it's not as simple as just pulling the disks from the mac and slapping them into my dedicated server because the disks are formated as typical mac extended journaled format. So I decided to do the hot foot/cold foot shuffle on my drives.
Pre-Step - Build Latest/Greatest FreeBSD on OS drive on Dedicated server and sync over the relevant bits - Out of scope for this post.
Step 1. - Source Server - BACKUP EVERYTHING to an external source (preferably multiple)
Step 2. - Source Server - Cleanly shut down Source Server, pull one of the two mirrored physical disks, and install it in the destination server
Step 3. - Destination Server - Boot, fdisk the new drive and create the zpool
Step 4. - Source Server - Boot it back up and snapshot the existing zfs partitions. I have a good number of these, so I wanted to make this quick.
So now I have snapshots of each zfs partition. Now for the beauty of ZFS's magic... I'm ready to start transferring data from my virtual disk on my mac, to the real mccoy on my destination server.
5. Source Server - Sync those snapshots directly over ssh to the destination server
Voila, checking the zfs list on the destination server, I can see the partitions being created and populated with block level data... A perfect copy.... Go get lunch, at nearly 500GB of data, this is going to take awhile, even on GB network.
After a refreshing lunch, and a short nap, I've got a physical disk with the replicated ZFS Volumes on the destination machine. Now I need to get the other physical disk into the new machine. I shutdown both machines, moved the physical drive over, and repeated the fdisk command above.
Now I need to add the new disk to the zpool
I ran a few zpool status checks and watched the first disk replicate to the second disk. After a couple of hours:
And there we have it kids. Now I have migrated a zfs raid pool from one server to the other. Just need to set up the sharing again
And we are done. Enjoy.
For the last couple of years (to save energy/$) I have been running my FreeBSD server as a virtual machine in Parallels on OSX... Why would I do such a thing? Where I lived last, the electric bill was absolutely outrageous and it had little to do with the consumption. Here in socal, each area is billed based on geographic location. They implement their own unique version of electric grid socialism this way by charging those near the coast (that rarely run A/C or any other high-draw utilities) approximately 1/3 more per kwh than people living further east. As you go east, down goes the per kwh pricing. It's more complicated than that, but you get the gist. It's one of the many "perks" I get for living in California.
So, to eliminate as many 24/7 power supplies and to reduce that bill, I moved from my dedicated server, to my mac pro (both of which run 24/7.) To do this and not lose the features that I so Love about FreeBSD/ZFS/etc..., I spun up a Parallels virtual machine and imaged (using dd) my physical server to the virtual server. The tricky part comes in moving from two physical disks to virtual disks on the mac. I have two 1TB drives in a zfs mirror configuration that source nfs on my network to all of our mac/linux nodes. So when i created the partitions in mac, I had to add a virtual disk to each of the 1TB drives. Anyone familiar with disk/os/filesystem performance is cringing now, but knowing what I'm doing and that I don't need high performance on this network, this was a perfectly acceptable solution.
Bottom line, I migrated my disks to virtual disks np, and made my freebsd virtual machine the internet gateway for the my network (including the parent OSX OS,) with a lot of fun networking/routing. Again too much to get into here..
Then I moved East a bit after buying a home (woot!) I'm not as interested now in running in this configuration, so I'm moving back to a dedicated server solution for a number of reasons, one of which includes home automation, more on that later.
So now I need to reverse this process. However, it's not as simple as just pulling the disks from the mac and slapping them into my dedicated server because the disks are formated as typical mac extended journaled format. So I decided to do the hot foot/cold foot shuffle on my drives.
Pre-Step - Build Latest/Greatest FreeBSD on OS drive on Dedicated server and sync over the relevant bits - Out of scope for this post.
Step 1. - Source Server - BACKUP EVERYTHING to an external source (preferably multiple)
Step 2. - Source Server - Cleanly shut down Source Server, pull one of the two mirrored physical disks, and install it in the destination server
Step 3. - Destination Server - Boot, fdisk the new drive and create the zpool
#fdisk -BI /dev/ada1
#zpool create data /dev/ada1
Step 4. - Source Server - Boot it back up and snapshot the existing zfs partitions. I have a good number of these, so I wanted to make this quick.
#cd /data/
[root@server /data]# ls -lha
total 2708
drwxrwxrwt 16 nunya wheel 17B Apr 29 12:36 .
drwxr-xr-x 23 nunya wheel 512B Sep 3 09:10 ..
drwxr-xr-x 3 nunya wheel 4B Apr 29 15:02 backup
drwxrw-r-- 10 nunya wheel 12B Apr 1 06:38 bin
drwxrwxr-x 11 nunya wheel 17B Jun 15 08:28 docs
drwxrwxr-x 5 nunya wheel 8B Apr 28 13:15 down
drwxr-xr-x 3 nunya wheel 4B Nov 22 2011 git
drwxrwxr-x 8 nunya messagebus 8B Aug 29 2011 jails
drwxrwxr-x 8 nunya wheel 10B Sep 26 2010 movies
drwxrwxrwx 831 nunya wheel 880B Feb 1 2012 mp3
drwx------ 2 nunya wheel 2B Apr 6 2011 mysql
drwxr-xr-x 15 nunya wheel 17B May 27 13:46 opt
drwxrwxr-x 13 nunya wheel 22B May 13 03:00 photos
drwxrwxr-x 2 nunya wheel 2B Jun 30 2009 projects
drwxr-xr-x 2 nunya wheel 4B Nov 18 2011 server
drwxrwxr-x 9 nunya wheel 9B Feb 7 2010 svn
-rwxrwxr-x 1 nunya wheel 2.5M Jun 28 2009 zfs.core
#for i in `ls`;do zfs snapshot data/$i@1;done
So now I have snapshots of each zfs partition. Now for the beauty of ZFS's magic... I'm ready to start transferring data from my virtual disk on my mac, to the real mccoy on my destination server.
5. Source Server - Sync those snapshots directly over ssh to the destination server
#for i in `ls |grep -v zfs.core`;do zfs send data/$i@1 |ssh server2 zfs receive data/$i ;done
Voila, checking the zfs list on the destination server, I can see the partitions being created and populated with block level data... A perfect copy.... Go get lunch, at nearly 500GB of data, this is going to take awhile, even on GB network.
After a refreshing lunch, and a short nap, I've got a physical disk with the replicated ZFS Volumes on the destination machine. Now I need to get the other physical disk into the new machine. I shutdown both machines, moved the physical drive over, and repeated the fdisk command above.
#fdisk -BI /dev/ada2
Now I need to add the new disk to the zpool
#zpool attach data ada2 ada1
I ran a few zpool status checks and watched the first disk replicate to the second disk. After a couple of hours:
server# zpool status
pool: data
state: ONLINE
scan: resilvered 422G in 1h32m with 0 errors on Tue Sep 4 19:27:23 2012
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada1 ONLINE 0 0 0
errors: No known data errors
And there we have it kids. Now I have migrated a zfs raid pool from one server to the other. Just need to set up the sharing again
#zfs set sharenfs=on data/bin
#zfs set sharenfs=on data/docs
#zfs set sharenfs=on data/down
#zfs set sharenfs=on data/movies
#zfs set sharenfs=on data/mp3
#zfs set sharenfs=on data/photos
#zfs set sharenfs=on data/projects
And we are done. Enjoy.
Monday, May 14, 2012
Picasa Duplicates and fdupes
I'm on Snow Leopard and was having troubles with picasa importing an nfs directory. Apparently in the last upgrade of picasa, it hosed my watched directories.
I found How to manually add directories to Picasa's watch list and it worked great... too great... As soon as I opened it up I realized that this directory had been partially imported by picasa before and I started seeing a massive number of duplicate photos.
With any duplicate removal, there is risk that two different pictures could share a name or two names could share the same content in one or more of the directories and finding the duplicate by name alone is not a solution for identifying duplicates. Enter fdupes. Fdupes analyzes the content of a file using an md5 hash and compares the hash between two files. If they match, they are truly a duplicate and can be dealt with accordingly. I did a bit of research on the best method of using fdupes because if you just run it as is, it will look for user interaction throughout this process. With large datasets, this isn't feasible. What I landed on came from Here.
If you're on mac and have macports installed (highly recommended if you're comfortable at the command line,) you can:
Having more than 10k files to check (many of them video,) my job took a WHILE
I found How to manually add directories to Picasa's watch list and it worked great... too great... As soon as I opened it up I realized that this directory had been partially imported by picasa before and I started seeing a massive number of duplicate photos.
With any duplicate removal, there is risk that two different pictures could share a name or two names could share the same content in one or more of the directories and finding the duplicate by name alone is not a solution for identifying duplicates. Enter fdupes. Fdupes analyzes the content of a file using an md5 hash and compares the hash between two files. If they match, they are truly a duplicate and can be dealt with accordingly. I did a bit of research on the best method of using fdupes because if you just run it as is, it will look for user interaction throughout this process. With large datasets, this isn't feasible. What I landed on came from Here.
If you're on mac and have macports installed (highly recommended if you're comfortable at the command line,) you can:
- Install fdupes
#port -dv install fdupes
- Backup Your Picasa Content Directory. For most this just means making sure Time Machine is up to date. For me this meant:
#rsnapshot -v hourly
- Now run fdupes against your picasa directory deleting indiscriminately.. (make sure you have a backup)
#yes 2 | fdupes -dr .
Having more than 10k files to check (many of them video,) my job took a WHILE
- Next I wanted a fresh start with my picasa database to make sure everything was in sync. I followed the instructions Rebuilding the Picasa Database
Subscribe to:
Posts (Atom)