Lost In The Mountains

Climate Model on a Mac #15: Watch those dynamic libraries!

2009-05-30T08:08:00.000-07:00

I recently upgraded my PGI compiler to version 8, and I had tons of trouble getting the climate model to compile and run. In this case, I decided to switch from mpich 1.2.7 to OpenMPI, on the hopes it would be better for the system and easier to set up.

However, nothing linked properly. If the software compiled, then it would have tons of MPI related errors. As a rule, I install all the libraries needed to run the model in the /opt directory, since it's easier to have more than one version of various libraries (different versions, different compilers, 32 vs 64 bit, etc), so I didn't think there could be a conflict in the libraries.

As it turned out, however, one of the MPI solutions still was getting installed in the /usr directory. On a mac, this can be a big problem because the macos goes for dynamic libraries in the usual places before loading anything in /opt. Hence, I was compiling for OpenMPI, but the OS was loading either mpich, OpenMPI without fortran support, or 32 bit instead of 64. The results of which are not pleasant.

The upshot to solving this problem is that I might be able to get the GCC/g95 solution to run the model instead of the Portland compilers. I'm happy with the portland compilers, but they're expensive to keep current. So, it's time to give it another go!

My first iPhone app: rejected by the app store! Annoying, but...

2009-05-27T23:19:00.000-07:00

I spent part of my morning fuming because my iPhone app was rejected. What made it worse was that it was rejected for false reasons. What made me a little more crazy was that there was no clear way to email whomever rejected the app a nastygram!

After I calmed down, I realized my reaction was in part because of all the other questionable rejections from the App Store. Moreover, I pondered how on earth they made the assumptions they made...

They rejected the app because when the app attempts to get data from the network, and the network is not active, there is no warning about the lack of connect. The annoying thing about this is that the application DOESN'T GET DATA OVER THE NETWORK! It doesn't use the network internally at all. All of the data is stored within the app.

Of course, after I calmed down, the obvious question arose, "why did they think that the data came over the network?" Looking at the screen shots they sent me suggested that what really happened is that they set some sort of search criteria (it's a database app) that resulted in no records being selected. Looking at what the screen showed when no records are selected was a little vague and could be interpreted that the data came over the network - it should have been clear that there were no records selected. Okay, my bad. I also decided to add alerts where a user should know there were no selected data, and when flipping a switch might cause such a selection problem.

Grr, now I have to wait another week to see if they let me in.

Cocoa in CBS' Numbers

2009-05-18T10:50:00.000-07:00

In the final episode of Numbers on CBS, Apple's Cocoa language appears on screen. Amita is kidnapped and forced to hack into a bank. On occassion, her code appears on screen and it's clearly Objective-C! Aside from the standard Objective-C notation, [object method], there are clear Apple Cocoa classes and methods, such as NSArray and addSubview. Not only that, but the code is available online!

See Safari Books Online!

So, it's not only Cocoa, but it's iPhone code!

I'm used to Apple hardware spotting on TV, but this is my first Cocoa spotting.

Update: As it turns out, the link above will get you to the right section, but does not display the source code. If you do a search on "// recover switchview" you'll then find the source code.

Doh! My first hard drive failure in my Drobo!

2009-05-01T08:38:00.000-07:00

It had to happen sooner or later. One of my hard drives in my Drobo failed yesterday.

As a rule, failures like these happen when there's a deadline looming. The deadline in this case is a talk my wife is giving tomorrow. Much of the imagery and data she needs are on the Drobo. The promise of the Drobo is that it should still be accessable even though a drive is out or during the rebuilding process. However, in this case, something about the rebuilding and the OS screwed up. Not only did the drobo become unavailable, the Finder on the OS locked up. As you can see from the screenshot above, the Drobo Dashboard is still working so I'm not messing with a reboot or anything until that completes. The OS itself seems quite functional under the Finder - it's just the Finder is locked up and cannot be forced quit. This is probably something related to how I set up my system rather than a failure of the Drobo or its software, but we'll see. Anyway, if there's anything else wrong with the rebuild, I'll post it here.

One thing I didn't see anywhere is how to figure out how the drive has failed. I need to look into this.

This all brings me back to the issue of adding drives in a cost-effective manner. Now that I have a dead drive, I have a choice to make. 1) Leave the bay empty, 2) Replace the drive with the smaller drive I removed earlier, or 3) buy a new drive.

Option 1 is a tolerable solution. Even with the loss of the drive, I would have a fully protected system with almost a half-terabyte free. The only issue is that there's not enough space for a rebuild should another drive fail; the data would be protected until such an event, but no rebuilding would be possible until a new drive is added.

Option 2 is tolerable as well. I can reinsert the 320 gig drive into the drobo and it would bump me back up to about 2 tb of useable space. However, at this point, the 320 might be better use as a backup drive that I store elsewhere. It's not large, but it would be good as a backup drive.

Option 3 is my preferred option, but will probably have to wait. At this point, if I'm going to add a new drive, I'd prefer the largest drive that I can afford, at least 1.5 TB. Although only 1 TB would be usable, rather than the full 1.5, it will allow some room to grow when the drobo fills up without having to buy at least 2 drives at a time. Maybe a WD green 1.5 TB drive....

My Drobo: A few months in...

2009-04-19T06:43:00.000-07:00

Well, I've had my drobo for a while now. I really have no complaints on the hardware; it works as expected and works well. I do have issues, however.

My first issue is my own behavior. I sometimes fail to RTFM. In this case, it was the part of the manual on formatting. I simply formatted the drobo from Disk Utilities. If I had RTFM, I would have learned that the standard format was for 2 TB volumes - if you stick more available space in the system, you'd get multiple volumes. The drobo can be formatted to anticipate a volume of 16 TB, which would have made more sense to me. The drawback to this format is slower boot times, but I'm okay with that. In any case, it's simply a matter of reformatting the drobo and start over. Unfortunately, I have the older USB 2.0 drobo and offloading the data took at least a day (and some struggling to find the disk space). Now, the drobo is reformatted and I've started the process of moving all the data back! ugh! take my advice, if you get a drobo, RTFM.

The second issue I'm having is planning my capacity. Now, Data Robotics provides a nice tool to help you plan your space, called the Drobolator. My current drobo has 2 1 TB drives, 1 500 GB drive, and 1 320 GB drive. That gives me about 1.6 TB of usable disk space. However, I wanted to add space to the system so I started trying to figure out the most cost-effective approach. It's not so easy. In a pure cost-per-gig view, I should buy 1.5 TB seagates. However, if I bought only one, the cost per gig went up considerably because the drobo would only 1 TB of space, rather than 1.5 TB. This is because size of the largest drive has to be eliminated from the total capacity for data protection should it fail (e.g. if you have 3 1 TB and 1 1.5 TB and the system was full, you'd be protected even if the largest drive failed since the system would use only 3 TB of the 4.5 TB). Based on the fact a 1.5 won't get me anywhere, I went with a WD green 1 TB drive (cost $10 more than a similarly sized seagate, but supposedly uses less power). Now this set up leaves me with 1 500 gig drive. This drive, when it comes time to replace it, makes more sense to swap with a 1.5 TB+ drive. At this point, the costs go way up (depending on how long you wait). Any drive I swap into the drobo now must be the largest capacity of all the existing drives, not just equal to the largest. If I swapped the 500 gig for a 1 TB, I'd get an additional 500 gigs of capacity. If 3 total TB available is enough, that's not a problem. But what happens if I need another 500 gig later? If all 4 drives are 1 TB, I'd have to replace a minimum of two drives: the first would become the largest drive defining the space for protection, and the second would then provide the available new storage. Try that on the drobolator, 4 1 TB drives and swap one with a 1.5 or 2 TB drive. At current prices, then, to add 5oo GB to a drobo of 4 1 TB drives would cost about $260, a shift from 11 GB/$1.00 to 1.9 GB/$1.00. That doesn't include of the cost of pulling out the 2 1 TB drives.

I have to wonder whether at this point it's better just to buy a second drobo and use the working drives you pulled out of the old drobo when you upgraded. Maybe, maybe not. I do tend to use hard drives for backups that I keep in undisclosed locations. If they are full, then I don't have any drives available. Of course, I could use a drobo for that purpose instead of simple drives. I'll have to consider my options if and when my drobo starts running out of space again.

Quick Review: Kindle on iPod Touch

2009-04-10T09:16:00.001-07:00

The Kindle application (Amazon) for the iPod touch brings the printed word to the iPod. Amazon has a great store for buying electronic books at making them available for the iPod. The question I had was: would I even consider reading a book on an iPod (or an actual Kindle for that matter)?

Once I installed the app, I took the plunge by stocking up on free books. Many public domain and some more current titles are available for free. And of those books, I wanted to read a book with and author and subject that was unfamiliar to me for something new. So, I tried "Persuader" by Lee Child. It has positive reviews on amazon (and apparently a very positive review by Newt Gingrich).

Of course the real question here is not whether the book is good, but did I read it? The answer is "Yes". Reading is surprisingly easy. The iPod Touch's Kindle interface was easy to use and ledgible. The number of words on a given page wasn' t large, so there's a lot of "flipping", but that's okay. I didn't have to fumble with a booklight. I didn't have remember where I put the book. I didn't have to make space to bring it with me. If I had a spare moment, I could pull it out and read a little. In many cases, a real book is a more satisfying experience (you can even get real book autographed), but there are also other considerations. For example, what to do with a book when you're done? I admit I'm a packrat when it comes to books, which is one of the reasons I don't read as much - I have to do SOMETHING with the book when I'm done. The Kindle experience allows you to delete you copy and allow you to download it again some other time if you want. So, the kindle approach will make its way into my life.

Now, the question becomes, will I ever BUY a kindle book?

First hands-on experience with the iPod Touch

2009-03-22T09:18:00.000-07:00

Well, I finally pulled the trigger on getting the ipod touch to start software development. I wanted to get it before starting development work so I could begin to understand how the interface, and system limitations, work.

I have to say that, initialially at least, I'm a little happier with the iPod that I expected. I never really had any hands-on experience with the iPod touch or iPhone, so I really didn't know what to expect in terms of user experience. The exception is that I'm a long-time mac user/developer.

I found that the interface is easier to use than I anticipated. So much easier, in fact, my two-year-old figured out how to use it only watching me for a few minutes (yes, I'm aware that the iPod and a 2-year old might not mix well). The keyboard was easier to use than I expected as well. I heard a number of complaints, and yes, I do make frequent errors (I have big fingers), but I'm surprisingly quick after only a day.

Some issues really cropped up in the UIs of some apps. For example, a free car simulator placed the steering control right next to the home button, and I kept kicking myself out of the game!

The most disturbing thing for me is the quality of the graphics of the system. The problem is that the graphics are surprisingly good - it's simply amazing that this thing fits in my hand. Of course, as a developer who doesn't have good design skills, this is a worry - the system sets up high expectations that I might not be able to reach on my own.

This software model for the App Store is also simply amazing. My neices all have Nintendo DS systems - about $130 retail (far cheaper than my iPod). The DS, which is probably safer to use for younger kids, is limited by the cartridge approach (yuk - I had to deal with that on my Intellivision back in the 80's). The App Store model provides tons of free apps (both useful and entertaining) you simply wouldn't get with the DS approach. Not to mention the difficultly around christmas time to find the desired cartridge that's been discontinued. Add to that the power (admittedly limited) of the internet on the iPod, the DS looks pale in comparison, at least from my adult eyes.

I just worry that I'm jumping in too late. Sure, I can expand out to Android and Palm Pre when it comes out, or maybe into the windows mobile marketplace. However, given that many of the niches are already filled, and my searching the app store suggests that small targeted apps might have trouble being seen, gives me a little concern.

The only thing to do is get on the diving board and take the plunge - right after I get my current programming work done!!

Check this out for your digital life!

2009-03-14T19:02:00.000-07:00

My Digital Life

A few minutes with Safari 4 Beta

2009-02-24T08:14:00.001-08:00

In case you missed it this morning, Apple released a beta version of Safari 4. Overall, it's very similar to the old version with a few notable exceptions. First, it added a "Top Sites" view, which shows a wall with the top 9 web sites you visit - showing a thumbnail of the page the last time it was visited (it shows a star for pages not updated recently -- at least I think that's what it means).

More interestingly, from this "wall" view, you can search your history and see the results in a coverflow format. The exciting thing about this search is that it searches content, not just URLs. Since I tend to like visual representations over textural reps, I like the potential of this search system.

The tabs have changed significantly. Instead of hanging from the bottom of the toolbar, it now sits on top. At first, I really didn't like this position, but I'm slightly warming to it. First, it goes against the standard mac window usage, where the top of the window is fixed and will lead to confusion. Now, instead of being able to simply grab the tab and move it around, you have to grab a little tiny bit of the tab on the right side to do the same thing. However, once you get used to that, it nicely allows you to move the tabs, rip them off for a new window, or move from one window to another. Another nice thing about the tabs is that it now shows a lock on the tab for secure pages.

There is also a coverflow interface with search for working with your bookmarks. This too has potential, but it limited by how often you loaded the pages.

I'll be happier, though, when 1Password gets up and running with it.

On the Importance of Metadata for Images and Movies

2009-02-11T22:58:00.000-08:00

One of the most difficult, annoying, and most put off chores of maintaining photos and images is keeping metadata records of some sort. In this digital era, the potential power of keeping metadata is enormous, but still difficult to keep up with a digital lifestyle.

Metadata is data about data. For images, that would include standard EXIF data, like camera information, and the who, what, where, and when of the image itself. Tools exist for attaching these data to images, less so for movies.

Why bother with metadata? Most people don't. However, the reality is that recording metadata isn't so much for you - the person who took the photo. The metadata is for other people. Most people know know the w's of the images they take already and assume that they will remember what the photo is about. Of course, your grandchildern won't know who those people are or why you bothered to take their picture. It's possible they won't care, but they certainly won't care if they don't know anything about the photo.

Over the last 4-5 years, I've been digitizing old slides and photos; mine, my mother's, and my mother-in-law's. It's rather painful to work through these images when there's no information. Sometimes there are recognizable landmarks, date stamps, or quick notes written on the back, but most info is very thin to non-existent. In many cases, the person who took the photo is no longer living, so there's no one to ask. One example was a photo of a tree and the only info written on it was "tree". It was apparently interesting enough to take the picture, but I'm sure that the people who took the photo knew why they took it... but I'll never know.

I suspect many of these images without any metadata will persist in our family for years to come until someone wises up and realizes that without the proper data, it doesn't mean much of anything.

Review: iPhoto '09

2009-01-27T18:21:00.000-08:00

Every time I think about abandoning iPhoto for something else, there's a new version that has something that appears compelling and I'm using it that much longer. iPhoto '09 boasts many new features, particularly Faces and Places. Faces is a facial recognition tool and Places is for geotagging. Both of these tools are facinating and compelling, but not without flaws.

The Faces feature really has two parts. The first part the location of human faces within a photo. This works fairly well, but it will miss many faces. Low resolution photos, or distance photos are problems simply because this tool requires a number of pixels to recognize that a space on the photo is actually a face. It also will fail more as a person's face is pointed away from the camera. The second part of Faces is actual facial recognition. Once the faces are located on images, you can go in and name the faces and it builds a neural network for each person so it will recognize that person in other photos. To work well, however, it needs a lot of training, especially you have a lot of photos and you're not a professional photographer who only takes great photos. Training involves a few approaches. You can see all the photos Faces thinks is a particular person, and you confirm or reject the recognition. You can also go into your photos and name the people on the fly. Either way, it takes time.

One question I have is when you manually define a face, and then assign a person, does it add that info into the neural network? I hope not. If Faces can't see the face, even though I assign it, I don't want it messing up the network.

Places allows you to set the location of you photos and allow you to sort through your photos by location. Assigning locations to photos is fairly easy. If you don't care about precision, you can set the location by place name. If you do care, you can use an address or use a Google Map to place the image. If you want to set the latitude and longitude directly, forget it. This is the major WTF issue with iPhoto '09. There might be a way to do set coordinates directly, but I haven't found it. I was worried that this was true in the Macworld keynote, and I'm disappointed. I didn't expect them to deal with track logs or those sorts of things, but I am not going to try to click/drag through Google Maps when I have the exact location. I need to find a solution that's compatible with iPhoto: Photoshop/Bridge won't do, sadly. What's with applications that allow you to set all sorts of metadata EXCEPT location?

An important consideration for these new features is whether they export the new metadata. Places does, in fact, add GPS and place name information to exported images if you so choose. Faces, on the other hand, does not export any metadata that I've noticed. This is a bit disappointing, too. However, it is possible to then tag all found images with the person's name and that would be added to the metadata.

It also appears that place names and Faces names are searchable in the media brower on iWeb (and thus for all apps using the browser). I searched for place names I know I added, but did not use a keyword, and they came up in the browser. Same is true for people I know I added to iPhoto. I did not see a way, however, to do a spatial search by latitude and longitude. You would have to use a city name, state, country, etc. if you want different spatial ranges.

With the exception of placing exact coordinates in an image, iPhoto 09 has a lot of potential power. However, neither of these new features are "free" in these sense that you don't have to do anything to take advantage of them. It's work, particularly if you have thousands of photos to work through. We'll see if the effort pays off.

Revamping my data protection plan

2009-01-26T12:20:00.000-08:00

Revamping might not be the right word, since I don't have a written plan, but I'm at least re-evaluating what I do to protect my data.

In the last week, I've been more seriously considering "cloud-based" data storage; that is, storing data on someone else's server out there on the "internets". The advantage of this is that if my computer is stolen, house burns down, or a tornado hits (there have been 2 near misses over the last few years), data in the cloud would be preserved. Thus, it can be effective off-site storage solution.

The other off-site storage solution I already use: put your data on some media and physically store the media off site. This is a great solution, but if something goes wrong, it might take a while to access the media to recover the data. "Cloud" storage, on the other hand, offers potentially instantaneous access to your data.

A problem with cloud storage (aside from cost considerations) is that you are dependent on the hosting company to maintain security and solvency (i.e. you don't want them to go under). Another problem is you are limited by bandwidth, either your own connections or the bandwidth allowed by the hosting company. This limits the practicality of "cloud" storage for some solutions. For a small set of documents, such limitations are minor. For many gigabytes of storage, this becomes a problem.

An attractive service is Amazon's S3. It's reasonably priced but the costs are not consistent from month to month. If I put 4 Gb of data on the site for a year, and never access it after that, it would cost a minimum of $7.60 per month (or about $91 per year). Mozy, on the other hand, is cheaper and unlimited at the pro level, costing about$60 per year. However, it is more of a traditional backup solution approach rather than just file storage (I say this without testing, however).

I'm planning to try Mozy for their free membership, but it will have to used as a backup solution for a limited amount of my data. My climate model results will have to remain a physical media offsite storage plan since the cloud is out of reach at the moment.

Martin Luther King Day

2009-01-19T11:11:00.000-08:00

Today is the day when Americans recognize the life of Martin Luther King. The man had a profound impact on this nation, and those impacts will echo throughout the future of America.

As a white guy, who was less than 2 years old when MLK was assassinated, from a small northern almost all white Ohio town, it took me a while to even begin to appreciate what MLK and like-minded people did for this nation. I doubt if I can ever fully appreciate it, but as I look out into the world and see ethnic and religious conflict I could see what America could have become.

A few years ago, the death of MLK touched me in a surprising way. While MLK was doing his civil rights work, America was at war. And when MLK died, my father was in Vietnam. Back then, there was no internet, cell phones, blogs, or any of those things. Along with letters, soldiers and their families would communicate using reel-to-reel magnetic tape. If you seen the movie Apocalypse Now, one of the men on the boat was playing a tape like that on the river. Sadly, these were often taped over or burned for security reasons. However, my parents saved a few. I spend some time converting these tapes into a digital form to preserve them before they fell apart. Most of the conversations that we have left are only interesting to my family. But to my surprise, my father very briefly mentioned the death of "Doctor King". He didn't say much about it, but it was there. In terms of MLK history, I'm sure this story is nothing much. I'm sure everyone had thoughts on MLK's death. However, it's impact on me was fairly strong because it was my father, my father's voice, and recorded during a war in a far away place.

Is Garbage Collection Needed For NSOperationQueue?

2009-01-06T20:25:00.000-08:00

I ran into a brick wall over the last few days. I've been processing my climate model images using two NSOperation subclasses and NSOperationQueue in MacOSX Leopard. In my case, the wall was the fact that the Queue simply refused to release my operations until the queue was empty. If I only had a few operations, this would be okay. However, I generate about 2000-2200 operations in a fairly short amount of time, and they should take about an hour or so total. However, when I tried submit them all, the system was completely bogged down by the end of the run and would crash. It was so bad, I'd have to restart if I wanted to get back to work.

If you're familiar with these objects, you'd think this is because I didn't release the objects after submitting to the queue. I tripled checked; everything was properly released. I stuck a NSLog statement in the -(void)dealloc method to see when it was called. Absolutely none of my operations were dealloced until the job queue was empty. There are dependencies, however, but eliminating those did not solve the problem and only added complications.

I tried a number of approaches to get the queues to empty. I created multiple queues, each one to be destroyed after running a number of jobs, but this set up failed. I created a queue manager that would feed a single queue after all jobs were submitted - failed. Frustration!

I then recognized that the problem was likely removing the completed object from an array was an autorelease issue. That is, the object isn't deleted right away when it's removed from the queue. Instead, it hangs around for a while until an autorelease pool removes the object. In my own code, this problem is easily solved by wrapping the code in a pool, between

NSAutoreleasePool *aPool = [[NSAutoreleasePool alloc] init];

and

[aPool release];

That usually solves the problem.

In this case, however, the queue is an opaque library class, I can't get in there and add a pool! In desperation, I started looking up the relatively new garbage collection in Objective-C 2.0. My hope here would be that garbage collection could be my way of injecting an autorelease pool action into the queue.

My first try was a failure, although some objects were released. I needed a mechanism to trigger collection. As it turned out, there was such a mechanism:

[collector collectIfNeeded];

I called this method when a change occurred in the number of operations held by the queue changed using KVO. This worked! I watched as the jobs ran and finalized!

As it turned out, however, if an object is a dependency for another object, it is still not released after it runs, only after the depended object is released, which makes sense. However, what this means is that a large number of objects still lingered far longer than they should. It appears that the queue takes a first come, first serve attitude that skips an operation if it isn't ready and doesn't come back to that object until every other object gets a chance. The solution for this (which failed without garbage collection) is a second queue that handled all the operations that were dependent on other operations. Thus, the second queue ended up running these operations far sooner and thus releasing both objects more quickly.

Finally, I'm ready to move on to the next problem!

Refactoring your code!

2008-12-30T13:36:00.000-08:00

My current programming project is Objective-C application for MacOSX to generate climate model results in the form of a web site and a PDF. It's actually a complete write of an existing application.

Why rewrite? The original code was built over many months, is stringy, and very hard to maintain. To get an idea of how hard, I've tried to rewrite this application five (yes, 5) times. The original application was designed to generate NCAR Command Lanaguage (NCL) scripts that generate hundreds of images based on climate model results. From there, the application generated makefiles that would continue the processing by running NCL for each script, convert the resulting postscript images into jpegs, and finally using FOP and Docbook to generate HTML and PDF documents with all the images and some related text.

There are several major weak points in this app. First, the conversion between postscript and jpeg images was originally handled by Imagemagik. Unfortunately, I've had a lot of trouble keeping this functionality working so I switch to generate javascript code to use Adobe Illustrator to do the conversions. In general, I was relatively happy with this solution (so much so, I gave up trying to get Imagemagik working). The unfortunate problem with using Illustrator for this work is that it no longer made the simple makefile approach to do the processing completely viable. I had to rewrite the makefiles to stop processing when the postscript files were ready, and allow running when the jpegs were ready.

Another problem with the code is that the makefiles and scripts were location dependent. In other words, if I moved my files to another location I would have had to regenerate everything if I needed to rebuild the image set.

The most significant problem was the Objective-C code itself. I'm a self-taught programmer, so I have a lot of bad programming habits. So, the code was stringy, repeated itself often, and easily broken. Thus, to make simple changes, such as changing the contents of the makefiles, was very painful. Hence, when I wanted to do something new in the project, I tended to opt to write a new version of the code... but it's a LOT of code to rewrite!

What's different this time? I realized I needed some new images in the reports. Furthermore, I needed a way to start adding more images as I came up with new ideas. Plus, I was just plain sick of the old code not being what I really needed.

The new app is not yet complete, but I've changed how I'm writing the code. The main thing is that I'm taking the time to decide if I can rewrite my code as I go by asking some simple questions: Am I repeating my code? Can I simplify the code by breaking out new methods? Can I join classes into a simpler class hierarchy? These questions are actually quite painful in many ways, often because the answer is "yes". When this happens, I break down and stop moving forward and see how I can improve the code. This often breaks existing code and can take significant time to propagate the changes through the code. In some ways, it is very much like the classic quote, "I would have written a shorter letter if I had more time".

However, the benefits have been huge. Instead of many classes repeating code, I now have a good class hierarchy where I need it with all common methods in the superclass. Instead of stringy code, I have much more clear and readable code. Still not perfect, but greatly improved. Debugging has become easier with better code isolation in methods. So far, I've simplified my original 14 NCL template scripts down to 4.

I've also learned a few new things about debugging. Exceptions are something that I've never fully understood in terms of when they should be used and when I watching for them in code. I've was hit by a NSMutableDictionary exception of trying to insert a null object. The problem is that I use a lot of dictionary calls in my code and the exception, nor gdb AFAIK, tells you where this occurs in the code. By implementing @try, @catch in most methods allows me to at least pinpoint the method. While not perfect, it certainly promotes the use if small and clear methods.

So, the lesson of this project is don't be afraid of refactoring. Do it as soon as the need arises. It will likely take up more time up front, but the payoff may include shorter overall development time, greater stability, easier readability, and extending the application is easier.

The Drobo so far

2008-11-13T12:17:00.000-08:00

With the Drobo up and running, I'm feeling a little safer. Right now, I have about 2 TB of disk space, which gives me just under 1 TB of available disk space. Of that space, I've used about half.

Overall, I'm very happy with the system. The largest drawback in this version of the Drobo is the USB 2. USB 2 is painfully slow compared to Firewire and SATA. However, in the way I'm using the system, the slow speed is tolerable.

The real test of the Drobo would be a hard drive failure. Sorry, I'm not going to even try to simulate that! Regardless, it will stay a part of the data protection plan for some time.

I voted! Now what?

2008-11-04T12:02:00.000-08:00

Okay, I voted this morning. It was uneventful with only a very slight wait (I purposely avoided peak times). The ballot was simple and, thankfully, paper. All simple quick and easy.

Now what?

Since I've done my patriotic duty, I have nothing else to do in this election but await results. The earliest poles close in 4-6 hours and no results will come in before then (which is a good thing, even though it's not instant gratification). So, what to do?

Well, my priorities are:

1. Burn DVD disk in preparation for a new Drobo.

2. Do some demo video podcasts of some software.

3. Work on a new audio podcast.

4. Satisfy some client needs.

5. Housework.

Bah, I know the truth. I'll probably flipping between political blogs and news sites to see what's going on.

Another test of my backup strategy

2008-11-02T06:57:00.000-08:00

It happened again! I had a hard drive crash. This time, the drive was a Seagate 1 TB drive about 60% full.

So what was my backup strategy and did it work?

I have three basic methods of backing up:

1. Time Machine - I use time machine to manage the backups of my laptop.

2. Backup to DVD - This method I was hoping to end. It's time consuming and hard to use with large groups of files. However, this has been relatively reliable.

3. Drive "mirroring" - Using tools such as rsync to maintain a second copy of a hard drive.

Method 3 is my latest approach and it's the system tested by my latest hard drive crash. Did it work?

Yes and no. I had a full copy of the drive where the data are safe. However, the backup itself was about 2 weeks old. As far as I can tell, however, about all I lost was a few downloads from the internet, including free iTunes songs and videos. So, I'd have to say the approach was reasonably successful.

One clear weakness of the approach was me. To make the system work, I have to do the mirroring far more frequently. Since I keep the mirrored drive offline when not backing up, this is not something I can do automatically (I have a LOT of data, so it's not practical for me to keep the drive online and active). So, I have to unmount another drive I'm using and add the mirror drive and then do the mirroring. This really can't change for me, so I need to make a regular schedule for now on.

The second weakness, which is far less obvious, is the lot of the hard drive. I worry that when I buy 2 hard drives of the same size and same brand, the drives come from the same manufacturing lot. Thus, if one is defective it's far more likely that the other is also defective (at least that's my fear). So, if one drive fails, as it just did for me, I worry that the other is also likely to fail putting my mirrored copy a high risk until I can get a replacement drive.

Given all these issues, I'm adding one more tier to my backup/data protection strategy. A Drobo. Drobo is essentially a low-maintenance RAID system (but not really a RAID). It holds up to 4 hard drives and allows you to mix and match drive sizes. If a drive fails, you pop it out and replace it. It is still likely to be vulnerable until replacement drives are installed, but the idea of not having to work about the timing of my mirroring would be a large improvement.

I've ordered a generation 1 drobo (since they're around $150 cheaper than the current model) from amazon:

And I'm already making plans to take advantage of the system.

However, this cannot replace the mirroring approach to backups. The Drobo will be used for more active projects and files, not long-term storage. Mirrored drives or DVDs will remain a part of my backup strategy for that work.

Right now, until I get the Drobo and replacement drives, I'm feeling vulnerable.

Book Review: Paul of Dune

2008-10-29T07:29:00.001-07:00

Paul of Dune is the latest book by Herbert and Anderson in the continuing expansion of the original DUNE series by Frank Herbert. Frank Herbert wove an incredible series of books, and I really enjoy the Dune universe. I like the complexity, and richness of the worlds, people, and politics. As such, I'm a bit of a sucker for more Dune books. If they write them, I will come. Paul of Dune is no exception for me.

Paul of Dune is set between the first Dune and the second book,
Dune Messiah. This new story fills in a little about what happened after Paul defeats Shaddam Corrino to become emperor and, in Messiah, when he known as a tyrant. Since this story is more of a gap filler, than breaking new ground in the Dune universe, it's more of a story of political intrigue than revealing anything particularly new. It seems to be more of a story about how Paul went from someone who had to be ruthless because of the situations he faced to someone who was pushed over the edge and became plain ruthless. Still, you remain sympathetic to paul, in part because as a reader, you feel that he's always forced along the path he followed, despite being a mass maker of war. This story is really about assassination and the consequences of assassination. In the original Dune story, there were attempts on Paul's life. In this story, we revisit assassination attempts that profoundly affect Paul and other members of his House and we are witness to new machinations hoping to rid the universe of Paul.

While I enjoyed this book, I found it a little lacking. I'm not sure that this book is intended to be a one-shot book or a new series, but it reads like it's meant to stand alone. As such, it seems to wrap up the story a little too quickly for my tastes. Also, link many prequels, you more-or-less know the outcome to the story before you read it - Paul is still emperor at the end - although it fills some gaps. Many of the other Dune prequels out recently take place long before the original Dune book, in some cases many thousands of years, and thus can stand on their own because much of what occurs in those books are mentioned only in passing in the original books. Nestling a book between two existing stories is much tougher to make compelling. Never-the-less, I enjoyed this book because I wasn't expecting anything radically new and, thus, I wasn't disappointed.

Richard Wright of Pink Floyd...

2008-09-15T15:55:00.000-07:00

Today marks the passing of one of the musical greats, Richard Wright. Wright was the keyboardist of one of my favorite bands, Pink Floyd. Tonight, I will be playing his "The Great Gig in the Sky" very loudly in his memory.

Story on CNN

iTunes 8 and Is HD Really All That?

2008-09-10T13:48:00.000-07:00

I downloaded and installed iTunes 8 yesterday and some of the new free HD programming available at the time of writing this entry.

It has been a long time since I've really looked at the preferences on iTunes. For example, I wasn't aware of the parental controls within preferences. It easily restricts tv, movies, and podcasts for ratings or explicit tags. However, not much is new, as far as I can tell.

A major addition is a new way to view your collection. Using the "grid" view, you can see a grid of images representing individual albums or groups of albums. It's a nice view, but I'm not sure if I'd actually use it. I tend to stick with the list view.

Another addition is "Genius", one of these fancy music discovery systems that takes what you like and find similar or complimentary music. In its main mode, you pick a song from the library, hit the Genius button, and boom, you have a playlist from your library of similar music. This seems surprisingly good at what it does. I've tried the process a number of times and found that I like the assembled playlists. I have a playlist that I developed using some online instructions to develop a playlist that acts more like a radio station, basing the playlist on my highest rated music and a sampling of more poorly and rarely listened-to songs. I don't think Genius will replace that function. But, on the other hand, I quickly discovered music I liked in my library I don't remember!

The second part of Genius is finding similar music in the iTunes music store. I noticed it produced a reasonably good selection of music, but I don't know if I'll be using that feature.

The big news, besides the return of NBC to iTunes, is that many shows will be available in High Definition (HD). Here, the content is 720p HD. HD to mean generally means a lot of money. In fact, I'm on the 2-year plan on upgrading. That means, whenever I think about HD, I'll upgrade in about 2 years (i.e. I really don't know when or if I ever upgrade my tv). Actually, seeing the HD content on iTunes made it easier for me to wait to upgrade. Sure, HD is nice, but I don't need it. Furthermore, HD TV shows on iTunes are huge: around 1.4 GB for 43 minutes of content. If I buy HD content at $3, I'm going to want to hold onto it and it will cost me to store that single file (probably 20-50 cents depending on the media). So, as of today, I have no plans on buying any HD content. Sure, I'll take the free stuff, so I wont feel so guilty about dumping it later. If BBC America allows Doctor Who content in HD, I might have a hole suddenly burning in my pocket, however!

So, I guess I'll continue playing with Genius, and watch the occasional free HD content. Other than that, for me, iTunes is largely unchanged, which isn't such a bad thing...

Whither MacOS 9 Classic: Time to update my data

2008-09-01T09:38:00.000-07:00

The following was a post I started a while ago and I'm not sure if I finished it. So, here's a quick re-write...

In practical terms, MacOS 9 is dead, again. Now that Leopard doesn't support "classic" mode, the Mac universe is going OSX... finally. On the other hand, Classic made me lazy. I have tons of stuff still floating about that is not OSX compatible. Now, I'm forced to do the unhappy task of data migration.

The Problem of Technology Creep and My Data

The one area in the whole computer/workflow yadda yadda is the problem of migrating data. When I say data, I mean anything, such as graphics, data files, video, etc. Many of these files are in custom file formats attached to specific software. When your software no longer works with the OS, well, you can see the problem. The technology creep, the slow continuous progress in technology, can cause data loss for many reasons, but the loss of apps is hitting me hardest lately.

Some Problem Apps:

MapInfo - I bought MapInfo (GIS Software) just before MapInfo cancelled the mac version. It served me very well over the years (I still used it up to Leopard). Now MapInfo must be retired. Because of business concerns, I now use ArcGIS. With Classic going away, I'm forced to do the migration.

MacDraw Pro - No, I don't use this anymore, but oh man it was stable in classic. A really well written app. However, I have tons of MacDraw images from my dissertation. If I want to keep them, they must be converted.

Canvas - Another graphics app, which just recently died on OSX, but my images were generated at a university with version 3.5, and I didn't have a copy!

Pagemaker - Similar problem as canvas. I just had a few files, but no app.

Corel Draw - I had a few files in this format. Again, I had no app to convert.

Climate model on a mac project: #14 Knowing when you quit...

2008-07-13T23:02:00.000-07:00

When not using a scheduler like Torque/PBS, it can be complicated to find out whether the model has quit. If the run was successful, you can have a reasonable idea when it SHOULD quit, but it might crash long before that time. As a result, you've lost hours, if not days, of computing time.

One solution to this problem is a handy application that comes with MacOS X: Automator. Automator is a simple way of getting the computer to do some repetitive tasks. In this case, the tasks are 1) run the model, 2) open Apple Mail, and 3) send an email. Automator makes these steps easy. The pre-requisite, however, that you have a mail account already set up in Apple Mail.

For step 1, you need the "Run Shell Script" action. Simply write the shell script to cd to the working direction and execute the run script.

For step 2, you need the "New Mail Message" action. Set the "To:" address, the subject, and message body to something meaningful. Also be sure you've selected the proper sending account at the bottom.

For step 3, all you need to do is actually have Mail send the message. To do this, you simply use the "Send Outgoing Messages" action.

To get this all to run; press play.

That's it!

Climate model on a mac project: #13 Post production!

2008-07-05T11:21:00.000-07:00

I've managed to complete a 100 year simulation for the Carboniferous. Over this span, the computer performance was, on average, 13.78 model years per day with a standard deviation of 0.13 model years per day using all 8 cores of the machine. I can't explain the large standard deviation. It is likely a combination of the computing activity of the software, hard drive interaction, the machine needing to do other work during the run, and heat. However, there is no correlation between run speed and duration. So, a direct connection between heat (using duration as a proxy, i.e. the longer the run, the hotter the machine) and run speed doesn't seem to exist. On the other hand, the longest duration run, over 6000 days, produced the slowest run speed, 13.5 model years per day. However, there is only one sample with such a long duration.

Now comes the post processing. I use a wide variety of "in-house" software to process the results and turn them into something manageable. I also use some free and open source software. One package is NCO (NetCDF Operators). Since I'm on a mac, there are at least three ways to get Unix/Linux apps on the machine. 1) Fink. a package manager, 2) Macports, another package manager, and 3) compiling, i.e. the hard way.

I've had no luck installing NCO using Fink or Macports. However, I used Macports to install some of the dependencies for NCO, such as ANTLR. So, compiling I go!

One thing I've found to be extremely handy when plowing through a difficult compile is to first build a script that you can set all your variables, switches, etc. so you keep a record of what you used.

Here's what my script ended up looking like:

#/bin/sh

make distclean

export CC=cc
export ANTLR_ROOT=/opt/local
export CPPFLAGS=-I/usr/include/malloc
export NETCDF_INC=/opt/netcdf/gcc/4.0.1/include
export NETCDF_LIB=/opt/netcdf/gcc/4.0.1/lib

./configure --disable-regex --disable-antlr --disable-shared

make

The malloc in the CPPFLAGS is there because many files could not be build without knowing exactly where malloc.h was located on the mac.

In any case, I only needed about one or two of the actual NCO commands.

Now, onto NCL (NCAR Command Language). NCL comes prebuilt for the mac, just install and go. However, it can be fickle. Today, it's just hanging on some files. We'll see what I can do to shake things loose.

Climate model on a mac project: #12 In Production!

2008-06-26T21:57:00.000-07:00

At last, the new machine is now doing production work and I've run almost 24 hours so far. First, it's running a bit slower than anticipated, about 13.5 model years per day. I'm not sure what's causing the slower speeds, but it's still an acceptable speed considering my original estimate was only 7 model years per day.

Temperature remains a worry for me. The CPU temps seemed reasonable, but the RAM temps seemed rather high for long-term processing. So, I ordered some fans to cool the RAM boards.

Tomorrow, I'll take a look at the output and start planning a possible Cocoa front-end for running the model.

Climate model on a mac project: #11 sysctl.conf and mpich

2008-06-23T08:08:00.000-07:00

A key component for running the climate model I'm using is MPICH. MPICH is an implementation of the MPI (Message Passing Interface) library for supercomputing maintained at Argonne National Laboratory (disclosure: I am a former employee of the lab). The climate model uses MPICH to break down the world into smaller parts and distribute those parts to each CPU. However, these parts must also talk to each other because climate information must move from processor to processor so that each processor knows what's going on around it.

MPICH comes installed in Leopard (I'm not sure if it's installed with the developer installation or the standard client installation) and is meant for use with GCC only. It is also likely that it's intended for XGrid as well. So, for this project, the pre-installed MPICH does not have the required fortran to run my simulations.

As it turns out, the default Leopard configuration is lacking a critical configuration for running MPICH on a single mac. It does not allow for shared memory. Using shared memory can speed up computation when you're using multi-processor or multi-core machines because it helps limit how much ethernet bandwidth the processes will use. Ethernet bandwidth is a major bottleneck for processing speed, which is why many high performance clusters use optical networking such as Myranet.

To allow shared memory on Leopard, /etc/sysctl.conf must be created. The contents of the file include something like:

kern.sysv.shmmax=536870912 kern.sysv.shmmin=1
kern.sysv.shmmni=128
kern.sysv.shmseg=32
kern.sysv.shmall=131072

Unfortunately, I don't know the optimal settings for these parameters, I just found these settings off the net. However, they do work.

And now, MPICH can be compiled and installed on the system. Here is the contents of my script to build MPICH:

make distclean
export CC=pgcc
export CXX=pgcpp
export FC=pgf90
export F90=pgf90
export F77=pgf77

export CFLAGS='-Msignextend '
export FC=pgf77
export FFLAGS=
export CCFLAGS=

./configure -prefix=/opt/mpich/pgi/7.2/ --with-device=ch_shmem --enable-g --enable-sharedlib

These instructions are for the Portland compilers and for a custom install location.

Unix, Microcomputers, 1985 on the Computer Chronicles

2008-06-18T21:30:00.000-07:00

Hindsight is 20/20, so they say. I was a big fan of the PBS series "Computer Chronicles" during my university days. I suppose that was partially due to the fact I didn't have cable and it was a inexpensive way of learning about what was going on in the computer world. In any case, I recently discovered that many of the episodes of this show were online. When I have a chance, I flip through the archive and watch a bit of the show.

Today, I found an extremely interesting episode from 1985 on Unix. Since Linux and Macs are both derived from Unix, I thought I'd give this one a shot. At the time this show aired, I was still in high school and I owned a commodore 64 and loved using it. I don't think I had any knowledge of Unix at the time. I think my sister and cousins had encountered Unix by then at universities, but hated it.

"Will unix become the next MS-DOS?" asked Stuart, the host. Apparently, at this time "microcomputers" were becoming powerful enough to run Unix. Speculating that Unix may become THE DOS of the future, the show goes into a brief description of Unix.

Interestingly, some of the terminology of the show is fundamentally different than today. Specifically, what's the difference between mini and micro computers? I don't know, actually, but I can guess there is certainly a price difference!

Also interestingly, some of the criticisms of Unix brought up in the show are still relevant today. Specifically user friendliness. The demo someone gives of Unix is so user unfriendly that even the presenter pounds on the keys a bit and then just talks instead. Later, he even says that not everyone needs Unix...

Another big plus was C, the programming language (I don't think I touched C until the late 80's early 90's).

Another surprise to me was that Unix was widely considered unreliable at the time.

Various forms of unix were discussed or mentioned, including a GUI-driven HP (HP-UX) version.

In an opinion piece later in the show, Unix is panned as a microcomputer OS... "a user-hostile operating system"

In the news bit at the end, and the Pentagon's biggest threat at the time was a "Mentally unbalanced 16-year-old".

And for Gruber at Daring Fireball, there's a news item on keyboards!

Watch the episode

Climate model on a mac project: #10 Rebuilding

2008-06-17T16:07:00.000-07:00

Now that the model clearly works using the test license from Portland, it's time to rebuild the system. So far, I've reinstalled MacOS onto the system and installed the developer tools.

Next, I installed required version control software. At this point, subversion and cvs are already on the system so I only needed to install GIT.

The next step in the process is to migrate my subversion and git repositories to the new system and get the servers going.

The version control systems will handle a few things. First, one of the systems will be used to handle the /etc directory. I found that I end up many different versions of mpich and other libraries and not only to I want to prevent messing up this directory, but possibly use branches to determine which versions of these software versions.

Model configurations will also be under version control. Right now, subversion is used to maintain these files. However, git will probably take over this role since a distributed model (since I use more than one cluster) make more sense for configuration management.

Next up: Modifying the system for shared memory runs.

GIT vs SVN

2008-06-16T07:40:00.000-07:00

Over the last few years, I've really taken to using version control for as many of my projects as possible. This includes more than just source code. Climate model configurations, unix /etc directories, writing projects, presentation work, etc have benefitted from using version control. When I haven't used version control, I've lost track of files and I made serious mistakes.

These days, I'm suffering from a wealth of options for version control. Originally, I did all my work in CVS. When subversion (SVN) was released, I quickly shifted my repositories and never looked back. However, many more version control systems have appeared and use radically different models for source control. CVS and SVN use a central repository from which everyone obtains their files and commits their code. The new thing is distributed repositories. That is, there might be a canonical repository, but the repository people use on a day to day basis may be somewhere else, like their own machine. The practical upshot is that you're not dependent the network for committing; you can do it locally. So, if you work on an airplane for example, you can still have the benefits of version control while you work.

I've begun using GIT for some of my work. Many of the concepts of version control are very different than for CVS or SVN, so the transition isn't the most straightforward. However, to give it a go, I used GIT to do versioning for my 2008 AAPG poster session. For this kind of work, I don't have to use most of the features of version control, but it helps keep me from generating dozens of versions of the poster on my system. As such, GIT worked great. The files are large and the commits are very quick. The overall experience was quite good. After a little more testing, I'll likely transition all of my writing/presentation work over to GIT.

GIT seems to have potential for climate model configurations as well since quick branching, and distributed nature could be quite helpful. However, using SVN, I keep on my model configurations together as one repository. In GIT, I'd probably would have to have separate repositories for model configuration families.

GIT also will likely replace SVN for my source code. But that will take a bit more testing before I start using it widely.

SVN, however, will remain in place for many of my projects. I don't see any version control system as perfect and SVN is still a very good system. Furthermore, git support on windows (when I do windows work) is limited.

Climate model on a mac project: #9 Getting in running in MacOS X

2008-06-03T15:33:00.000-07:00

One misgiving about running a climate model on the mac is that I don't trust linux to handle the Mac's fans to keep the mac cool. Apparently, there is even more to it. The mac also will throttle memory bandwidth if the ram gets too hot (see technote TN2156). Whether or not a Linux install not targeted to a Mac would handle these issues, I don't know.

All those worries seem to be moot, however. I managed to get the model running within Leopard. The next few posts will concern some of the details of how this was accomplished.

Tom

Climate model on a mac Project #8: Memory Upgrade...

2008-05-31T08:39:00.000-07:00

The minimum ram requirement for the climate model was 2 gig, which is what came with the computer. However, I decided to get 8 more gig for the system. At first, I left the original 2 gig in the machine (arranged according to the docs). However, this caused an enormous performance hit: the system was 12% slower running the model. Dropping the original 2 gig brought the speed back up, more or less, to my original performance figures. I'm still not sure that this was a memory arrangement problem or a mixed-card-size problem...

Tom

The Retirement of the USS Kitty Hawk

2008-05-28T07:33:00.000-07:00

The USS Kitty Hawk, the aircraft carrier my father served on during Vietnam, is headed toward retirement in Washington. My father's old squadron, VA-52 Knightriders, has long been retired along with the A-6 Intruders they flew. It seems this era in my father's life is being slowly moving into pure history. It's all making me feel a bit nostalgic today...

Climate model on a mac Project #7: Performance continued

2008-05-24T16:49:00.000-07:00

After attempting a slightly longer R15 run on the model, the performance stayed about the same. It runs at approximately 72 model years per day. I was hoping for more, but that's quite respectable.

Currently, I'm running my first T42 simulation. I was guessing I would get 7 years per day for this run. I was using a fairly linear CPU comparison with Gondwana, my older cluster, which uses 16 2 ghz Opteron processors, comes in just under 10 years per day.

Well, initial performance results say that my calculations were way too low. The model produces a month of data every 7-8 minutes so far... that translates to a year-per-day range of 15-17. This result was far better than I anticipated!

We'll see if these results hold...

Tom

Climate Model on a Mac Project #6: Success!

2008-05-23T22:35:00.000-07:00

After a long struggle trying to work with various compilers, today, I decided to try the Portland C/Fortran compilers. I've had success with them before. Fortunately for me, it worked! I actually got the model running at a low resolution! Tomorrow, I'll try a high resolution.

As of tonight, I've only run a single year at the low resolution. I have 8 processors, and only 2 gigs of ram. It took about 20 minutes to run the single year. Translated out to a full wallclock day, that's about 72 model years. I have to admit I'm a little disappointed in the speed. However, I am a little thin on the RAM, with only 2 gigabytes, which probably slows things down. CPU usage didn't seem to peek over 50%, so more ram may translate to more speed.

Once I looked at some comparisons with other system, I was a little happier. Take a look at this figure modified from that FOAM homepage:

Rodinia is my machine.

The 72 years-per-day speed, if it holds, compares very well with existing, high powered clusters. Of these systems, I've had some experience with JAZZ running a higher resolution model. Given that Rodinia's performance is twice as fast as JAZZ, I feel the higher resolution models will perform quite well on this system. It's also important to note that these other systems listed are not newer clusters, although many are still in operation. On the other hand, this system has a relatively low barrier of entry, it can double as a desktop mac when not in operation, and most importantly, no competition for cpu time.

I still have more configuration work to sort out. Getting the model to run via PBS on a user's account is the priority.

More details soon!

Tom

Climate Model on a Mac Project #5: Installing additional software

2008-05-14T08:38:00.000-07:00

The climate model I'm planning to run is compatible with the g95 compiler, not gfortran. So, first thing was to download and install the g95 binaries.

The next step is to download and compile the NetCDF libraries from NCAR. Before configuring, I had to export the FC environment variable to g95 to get it to compile correctly with g95. Running make check also passed.

Other tools (at least for me) that are needed, but won't describe is svn, git, and nco.

Climate Model on a Mac Project #4: Installing ROCKS

2008-05-14T08:12:00.000-07:00

ROCKS is an NSF (National Science Foundation) supported project to develop and maintain cluster environments so they provide cluster installs for a wide variety of platforms. It contains the usual things, like PBS, ganglia, and many others.

To install, the first thing was to add a new SATA drive to the machine. I had a 250 gig lying around so I added it to bay 2 on the mac.

Installing rocks was easy. The procedure was relatively straightforward. One problem that turned out to be a non-issue is that Rocks said it would install itself on the first drive (I thought that would overwrite the macos on the first drive, and leave the drive in bay 2 blank). As it happened, it installed on the blank disk. All is well.

On reboot, it appears all the critical elements of the system was running: ganglia, PBS, etc.

Now, on to the hard part, getting the model working!

Climate Model on a Mac Project #3: First snag

2008-05-14T08:07:00.000-07:00

Yesterday, after bringing the machine to my office, I ran into the first snag. It wouldn't boot! It would boot to a "blue screen" and die. So, I headed back to the apple store and they tested the machine. Annoyingly, it worked fine for them. It turns out that I was using a cheap 3rd party flatscreen with a DVI to VGA converter. Sadly, this wasn't good enough for this machine.

Luckily I had an older Apple display that used the old ADC connector. For $99, I picked up a adapter and now the machine runs just fine. An annoyance, but at least the machine wasn't DOA.

Climate Model on a Mac Project #2: Unboxing

2008-05-14T08:05:00.000-07:00

I've posted unboxing photos at my personal website.

Climate Model on a Mac Project #1

2008-05-14T07:55:00.001-07:00

Computer clusters can be expensive to buy and expensive to maintain. My current linux cluster has 18 CPUs, and takes up about 13U of rack space in a colocation facility, and uses 17-18 amps of power.

The current line of Apple Macintosh computers, on the other hand, have up to 8 cores at 3.2 ghz per machine. While even at this configuration, it technically is not as powerful as my large cluster, it is far cheaper to buy, costs much less to run, and can double as a powerful desktop. It also would have the added benefit that it would not have to use ethernet to communicate between processors (a major bottleneck).

My back of the envelope calculations on performance is impressing. For the model I'm using, my cluster yields just under 10 model years per day of running. For this machine, based on CPU speed alone, it should produce 6-7 model years per day. This calculation could over/under estimate the performance in key respects. First, the cluster uses opteron processors, and the mac is Xeon processors. In theory, the Xeons should be a little slower at the same clock speed. Second, since the mac does not have to use the ethernet to communicate, the system could be much faster since ethernet tends to be a bottleneck on performance. Time will tell, assuming the model runs on this platform.

I'll be using this blog to document the setup and performance of this machine as the system is put together.

Book Review: Harry Potter and the Deathly Hallows

2007-07-28T23:25:00.000-07:00

We all knew it was coming: the end of Harry Potter. The final chapter in his saga is an interesting story. For a while, I kept confusing Harry Potter with Frodo from the Lord of the Rings since there are surprising similarities; lots of travel running from the enemy, lots of encounters with evildoers, a quest, and other general hardship. Even the end had a similar feel, at least to me, to the ending of LOTR.

This is not to say that this HP is a rip off of LOTR, in fact, these elements are common in fantasy writing and probably make everyone think such works are LOTR rip offs! However, there is enough uniquely HP to make the story clearly its own. In some ways, this HP is a witch's brew of all the HP series brought to a nice boil in a cauldron over a hot fire. Almost everything that occurred and all the people from the previous stories is mentioned or makes an appearance. Of course, when a writer is ending the series, you have to tie up all the loose ends (unless you want to leave the door open for future work).

The ending of this series wasn't that much of a surprise. Of my major predictions of what would happen, several were dead on, one was almost right, one was wrong. Not bad. Despite my good predictions, there was several elements I didn't predict. I wont get into details too much, but suffice to say that all these kinds of events have serious consequences. (This is what I like about the new Doctor Who series, his companions leaving for travels in the TARDIS can leave hardship and heartbreak for those left behind). Just who lives and dies at the end of this series, well, you'll just have to read it yourself.

If you like HP, you should wrap up the series with this book. If you're a younger reader, you might want to wait a bit.

WWDC Steve-Note 2007 - The good about disappointment

2007-06-12T18:41:00.000-07:00

Well, Steve Jobs gave another great presentation at WWDC this. As usual, people are a bit disappointed about the announcements regarding MacOS X Leopard.

The announcements (updated finder, ical, 64-bit, etc etc) are boring. For the consumer, it SHOULD be boring despite Apple needing to find a way to make it exciting. Now, I understand we don't want it to be boring - we want a lot of new toys for our $129. However, operating systems should not be too much about the bells and whistles. I heard some podcasts reviewing some of these changes and they were disappointed in things like how the finder works and how they'll continue to use 3rd party apps. As a developer, I say "that's wonderful! Go ahead and buy those third party apps". OSs have to be the lowest common denominator AND there has to be room for developers to extend the system for power users. So, I'm fine that some people think it's not enough. But then, that's not everyone.

On the other hand, the most important thing about an OS is how developers interact with it. Now, that's cold comfort for the consumer, but it's true. Most of the features announced has two purposes: marketing to consumers, marketing to developers. Marketing for consumers is always going to be weak when it comes to mature and maturing OSs. Look at Vista. But both Leopard and Vista show off much of the 'under-the-hood' tech in their everyday use. That is for the developer.

Never-the-less, I think Leopard is shaping up and I look forward to it's release come October and I will be happily upgrading.

Disaster Recovery

2007-02-26T21:27:00.000-08:00

My big fear is losing all my data in some sort of system crash. Well, I just had a system crash last week (bad hard drive). As it turned out, it was marginally painful. My backup system was designed for managing my data. Fortunately, my data were on the drive that survived. The dead drive housed the system OS.

After some false starts, I managed to find an old copy of the system and restored it onto a new disk. Within a few hours, the system was back up to normal, more or less.

Although I didn't backup as religiously as I should have, even a marginal backup scheme prevented a total loss. Any backup is better than none.

SATA vs. Firewire vs. USB 2.0

2007-02-13T10:56:00.000-08:00

’ve been needing a serious evaluation of my backup habits. One area is mixing my storage media for longer-term backup. Up until now, I’ve been using DVD’s, but given the hundreds of gigabytes of data I can create in a month or two, and questions regarding long-term storage of writable DVD media, I’ve begun delving into the possibility of using hard drives as part of an overall strategy. But the world of external drives has become exceedingly complicated. Firewire (IEEE-1394), USB 2.0, and more recently SATA/ESATA.

The performance rating of these interfaces are 400 mb/s for 1394a and 800 mb/s 1394b, 480 mb/s for USB 2.0, and 1500 mb/s for SATA 1 and 3000 mb/s for SATA 2. But what are their real-world performances like? The following is the first part of my exeriments on performance

The equipment:
1. The computer - Macbook pro - USB 2, 1394a, Express card for external SATA, internal SATA drive

2. External Firewire/USB 2.0 drive with a 250 GB IDE drive.

3. External SATA 1/USB 2.0 drive case with a 320 GB SATA drive.

Method:

I used a typical directory for me, 17.35 GBs, and copied from one drive to the other several times using a shell script. Although I’m mac based, none of these files required resource forks so no special handling was required.

Here are the results of copying back and forth to the mac’s internal drive. The figure below shows the copy time in seconds (best results are SHORTER).

Not surprisingly, the SATA interface performed the best. USB 2.0 performed the worst. Generally, the hardware limitations of the mac’s internal workings are a major factor here, but the results are inline with other results out there on the web. The other limitation is the external drives themselves. Keep in mind that most drives can only write about 30-80 MB/s and the best performer here is only about 230 mb/s, or about 30 MB/s (note that this speed is far far slower that each of the interface’s ratings).

The second experiment is copying external drive to external drive. Since I only have two cases and limited USB 2.0 ports, I was limited in the experiment. Here are the copy times in seconds (best results are SHORTER):

Again, USB 2.0 is the slower of the interfaces. Interestingly, the copy speed to the SATA drive from the Firewire drive is relatively fast: 293 mb/s, or about 37 MB/s; this is the fastest copy speed of the experiment, and it was approaching the 400 mb/s for the firewire interface. I consider that a very good result for Firewire along with SATA.

This first experiment show that my purchase of a SATA1/USB combo case was probably not the best solution - a case that had SATA1 or 2 and firewire would have been far better for my needs. USB 2.0 as a solution is clearly not good enough for me.

Okay, I understand - XCode, Universals, and Version Control

2007-02-13T10:49:00.000-08:00

Originally published: Monday, August 21, 2006

Will Shipley of Delicious Monster argued at a group discussion at Adler Planetarium that developers should avoid using their own custom frameworks, and harness the power of version control instead. To some extent, this didn’t go down all that well. As someone who was saw the discussion online after the fact, I was left a little perplexed, but then I do consider myself an amateur. After all, frameworks are handy ways on reusing code, and how would a version control system going to help me reuse code effectively?

Enter my work to port my applications to Universal binary (ppc/intel binaries). I’ve had to go back and retouch all my frameworks, and figure out how to build universal binaries of the open source code I use. The universal open source code began to show me the Will Shipley light...

Usually, open source code compiles easily; run ./configure, and make. With universal code, you have to do a bit more than that, depending on the project. So, it was time to get the open source in a local version control repository. The problem is, of course, at this point I now had about five separate version controlled projects. What I typically did before now is check out and compile each project manually. The problem with this is the code can easily go stale and I can make mistakes easily. Furthermore, it becomes a pain if I need to switch computers, which happens more often when I travel, and I have to go through that entire process again.

Enter subversion. Subversion is my preferred version control system. Of the many things in subversion I don’t know, the svn:externals property is one of those things that would have made life much easier. By setting the svn:externals properties, subversion can go out to other repositories or other places within the same repository and bring them into your project hierarchy. So, all those frameworks and open source projects can become a part of my main project fairly easily. Then it’s just a matter of making my main project dependent on those sub projects. No tedious copying, and no working builds going out of date.

So, now that I can do all this with libraries and frameworks, can I get rid of my frameworks?? As it turns out, I can, in theory, get rid of one of them (well, actually all of them, but let’s not get greedy), although I haven’t as of yet. All it takes is reorganizing my existing framework repository to just include the files, not xcode project, and check them out within another project. Then, they act just like regular files I can add to a project - and updating as needed!

So, now that I see Mr. Shipley’s point and how it can be done, will I take his advice? Time will tell.

Universal Mac App: My first shot at Universal compiles

2007-02-13T10:48:00.000-08:00

Originally published on Monday, August 14, 2006

With a good deal of trepidation, I grudgingly converted one of my apps to Universal format for Mac. That is, it runs on native PPC or Intel code. Although there has been a great deal of comment out there that building Universal apps is easier than it sounds, I felt my app was one of those which would be a difficult battle. The app itself uses GUI, open source, a custom framework, and byte order specific code. Worse still, my code style isn’t what I’d call “good”. Uhg!

For the most part, it was actually very painless. Fortunately, the open source code handled byte order issues internally, so that turned out a little easier than expected. In the areas in which I specificially swapped bytes, well, I mostly had that code right anyway, so with a little work, it’s just fine too.

Of course, I haven’t fully tested the result. As far as I can tell, it works almost exactly as bad on a intel mac as it does on a PPC mac. When I say bad, I mean that it’s not well or fully implemented. It’s just a GUI app I’ve been working on for years that I never bothered to polish (I plan on making a polished version, but will I???).

So, rating my first experience for 1-10, I’d say 8.5 (1 point off for having to do it in the first place).

XML and InDesign

2007-02-13T10:47:00.000-08:00

Originally posted Saturday, July 8, 2006 on another blog site;

One of the issues I’ve been wondering is how well does InDesign really deal with XML and can Docbook be used directly? So, what’s wrong with docbook? For HTML creation, it’s great. For PDFs, it’s good. Compared to InDesign, the PDFs are not higher in quality, but the fact docbook is a free system is nothing to sneeze at. What’s forced me to move, in part, from Docbook to InDesign is because InDesign allows placement of postscript images, where Docbook-FOP does not.

One of the surprising differences, particularly in XML handling, is that InDesign doesn’t handle Docbook out of the box. It can be done, however. But for my purposes, I’m not really THAT tied into Docbook because almost all of my Docbook files that I’m currently using are generated by software. With some changes in the software, a little setup time in InDesign, and a javascript for InDesign, I also automatically layout the same information (with postscript images instead of lower quality jpegs) in a much nicer form and, I think, an easier to control environment.

Where InDesign falls flat in comparison, at least so far for me, is the exporting to GoLive and generating web pages. Not happy with that workflow, at least not yet. Docbook handles this much better. So, I’ll still use Docbook for html layout.