It was encouraging for me that Tom Kyte’s keynote address quoted my definition of Best Practices from my 2001 Sun Online Blueprint “Sun/Oracle Best Practices”, which is a definition I still stand by a decade later.  I sometimes define Best Practices as being “the things you have to do yourself until the software evolves to do them automatically”, but I’m adamant against the term being diluted to mean “some clever things we do or did that we believe are part of our success”.  I also liked Tom’s candor about how even the best feel “stupid” when encountered with a problem they’ve never seen before.  I liken that to my own metaphor about fear of bears and the value that can bring to one’s bear-hunting outcomes.

Although I missed making a timely submission during the original call for papers, the good folks at Hotsos latterly signed me up as a standby presenter.  Since Henry Poras was unable to present his scheduled session on “Diminishing Resource Utilization and Saturation Limits Using AWR History and Queuing Theory” on Tuesday afternoon, I got to present my “CPU QoS (Quality of Service)” talk in that time slot.  My session was very well-attended, and I received nothing but positive feedback regarding the material.  I am passionate about the qualitative aspects of CPU usage, and I hope I motivated some attendees to go exploring in those dimensions.  Ironically, my session in Hall B ran concurrent with “Performance and Sizing of Oracle on VMware” in Hall A, which was presented by my esteemed former colleague at Sun, Richard McDougall, now Chief Performance Architect at VMware.  If it had been a Sun-centric conference rather than an Oracle-centric conference, Richard would have drawn an overwhelming share of the attendees!

Hotsos organizers chose Disco as the theme for Tuesday night’s dinner party, and I saw no choice but to attend in costume as The Disco Duck!  Yep; I was in Atlanta when Disco was big, and I felt I just had to make a statement!  Younger persons and attendees from overseas mostly thought I was a chicken, but the DJ set everyone straight.  He made an on-the-spot iTunes purchase of the original 1976 Rick Dees’ Disco Duck track on his iPhone – which led to my command performance on the dance floor.  A thousand thanks to the kind lady who joined The Duck on the dance floor; I believe she would prefer to remain anonymous!

On Wednesday, I had the pleasure of co-presenting with Tanel Põder on “Understanding LGWR, log file sync Waits and Commit Performance”, with my slides filling in the part about what happens to I/O requests once they leave LGWR to run the gamut of the I/O stack to be serviced by the target hardware.

Tanel Põder’s training day on Thursday was incredibly well-attended, and I’d say he proved his point that (paraphrasing) “The Oracle performance tool of choice for the 21st century is … SQL*Plus!”.   The speed at which Tanel navigates Oracle is impressive, and he does a great job of empowering others to do likewise!

Marco Gralike captured numerous video clips from the conference, including my brief tour on the dance floor as Disco Duck on Tuesday night.  Google “Hotsos 2010″ to find these and catch a glimpse of some of what went down in Dallas last week!  The Hotsos Symposium has become my favorite annual outing!  I’m already planning for Hotsos 2011, where I hope to present a session tentatively titled “Brute Force Parallelism”, the story of a rare 100X speedup payoff we got by adding a little C code with some PThreads logic to a monthly procedural batch-processing job that originally took far more than a month to run.  Yep; I need a good excuse to write that one up properly and share it — and try to get on next year’s agenda well-before the deadline!

In my almost 13 years at Sun, I was blessed to learn quite a lot about what goes wrong with computer systems performance and with the processes and politics that arise when things don’t work.  I played a major leadership role in Sun’s internal communities focused on performance and Oracle, and was the driving force in the creation of the Sun team that handles performance-related service calls.  I cherish the many amazing – and sometimes confounding -  colleagues and customers with whom I’ve had the privilege of working.  I now feel like I’ve graduated — with honors!  In my “post-grad” work, I will continue seeking opportunities to apply and extend my knowledge and skills in the areas of computer systems performance and capacity, and I will continue to be active in forums such as the Computer Measurement Group (CMG) and the annual Hotsos Symposium.

For years I’ve fought the notion that “premium talent is too expensive” in industry – though everyone of that opinion always seems to expect the very best medical care if they themselves ever happen to end up in an Emergency Room!  When the stakes are high, experience counts!  After years of “smoke jumping” for Sun, I’m now feeling like an IT analog to “Doctors Without Borders”.  My business model for consulting will be simple; leverage premium experienced resources for triage, diagnosis, strategy, and training – and thereby empower IT organizations to succeed using their own wits and staff talent.  Between myself and my broad network of expert associates, I believe I’ll be able to help some companies find their way out of their performance and capacity miseries, and onto smoother waters.

[ Originally published at http://blogs.sun.com/bobs on Friday Feb 09, 2007 ]

For your reading pleasure, today I’m publicly posting a paper I wrote back in 2001 called “Oracle I/O: Supply and Demand“. This paper was delivered at our Sun User’s Performance Group (SUPerG) conference in Amsterdam in October of that year, but never made widely available to non-attendees. Though the paper is a little dated, it still represents my approach to explaining I/O in general. That is, what you measure with Oracle, iostat, or any other tool – will be the result of some demand factors constrained by some supply factors. Sounds simple, eh? I’ve found that illuminating both supply and demand factors – and showing how they interact – is the key to untangling the physics of any given situation. While the principle is simple, “the devil is in the details“, and those details are continuously in flux!

If I ever write a book, this will be the basis for a chapter in it! I hope that making this paper available here will help others further their holistic understanding of disk I/O in general! This paper could be enhanced by discussion of other factors, including NFS, ASM, and RAC/grid factors for example – but I’m more likely to blog about such things than take a stab at a Grand Unified White Paper in the near future.

I’ll never forget that trip to Amsterdam! It was just after 9/11. Air travel was rather tense, to say the least. SUPerG attendance from some parts of the world was severely impacted. Sadly, in the time since, Sun’s SUPerG program has wound down. I have many fond memories from my SUPerG outings!

I’m proud to be part of the Performance and Applications Engineering (PAE) group at Sun Microsystems, where my clever colleagues continuously amaze me with their creative and diverse means of tackling the most challenging performance analysis tasks. My work centers on the performance of Sun’s products and performance-related service delivery, and my activities are primarily clustered under our “Customer Focus” programs.

My traditional technical focus has been in the areas of Oracle and storage performance. In recent years, I’ve been quite involved in matters of service delivery where performance is involved. Since much of my career has consisted of “smoke jumping” exercises (a common industry term for flying into difficult situations and sorting them out), I’ve seen that most of what goes wrong in production are repeated patterns. After several years of flying around, I finally figured out that it is far less wearisome and higher-leverage to capture knowledge on paper than to wrestle cases one-by-one, so I turned to writing a while back.

Among my easier-to-find publications are the Sun Blueprints Online^TM publications “Sun/Oracle Best Practices” in 2001, and “Performance Forensics” in 2003. I also develop material for various performance-related training internal to Sun, and contribute to selected industry performance symposia.

In today’s fast-evolving technology world, knowledge rots so quickly that is can be discouraging to try capturing it at a moment in time via whitepapers. Therefore, beginning now in January 2007, my Good Intentions are to make the occasional posting here in my blog or contribute some tidbits here and there on solarisinternals.com. Nobody should hold their breath waiting for me to post anything, but as of today, I’ve made a place in corporate cyberspace where my time-to-market can be minutes, not weeks or months!

Whether or not I span the generation gap and start blogging about my kids, cats, cars, or personal concerns is very much TBD!