darrelhuish.net

I recall double checking the fit of my pastel blue tuxedo, (can that memory really be right? pastel blue?), and wondering how many of our guests would arrive. There was always that tiny doubt that my dad might not make it, and a good possibility that Mike Flores would arrive late with a tall tale of an excuse. (He was right on time, and dapper too.) I was asking (once again) for a rundown of who would say what, and go where, and when. I was having trouble (once again) absorbing if I was supposed to walk down the aisle by myself, or with Manny, or just what. “Could we go over that again?” I suppose the idea of an actual rehearsal wasn’t en vogue in Douglas Arizona in 1975; at least not when the wedding was soon to be held in my parents living room. Who among those that were there have bet even money that 33 years later, those two teenagers would still be married. I wouldn’t have bet, not because I didn’t have confidence, but because I couldn’t quite grasp that much time passing, and certainly had no thought of ever being 52 years old. Even now, it’s a surprise to realize that my seemingly old, and somewhat fallible, parents were not even 40 in 1975. Yes, I was a kid, and they were too. Wild.
What are the odds that someone that was at the wedding that afternoon will read this blog? Doubtful. Extremely so. Still, here is an expression of thanks to all of you that suspended your disbelief and helped a couple of teenagers to start married life. We are all connected in ways we don’t know — and we never know when a moment of truth, or a fork in the path, is just a little bit up ahead.

Cheers!

Regular readers here will know that I have a tendency to be optimistic; look for the silver lining; shovel into a pile of manure, saying: “There’s gotta be a pony in here somewhere!”

This week we decided to move the go-live date of our SIS implementation. Here is a good summary explanation from another source: Load testing was conducted this month to test the system’s capability to handle the student information demands the colleges will generate, and the results were not sufficient to meet our needs. Certain other aspects of the new system were testing insufficiently as well. Therefore, the system will continue to be enhanced in the ensuing months to ensure a successful delivery next spring. We simply cannot compromise our ability to meet the needs of our student population, particularly during a high-registration month.

And so we decided to move the date.

And so you may ask, “where’s the pony?” The pony is that Maricopa made a wise decision not to gamble with the student experience. There is no doubt we could have brought the system up as scheduled, but with some real risk of significant disruption to students and faculty. Our egos might have said “yes, we can do this. Let’s go for it!” but the compassionate voices were at the forefront and said “let’s not make it harder for our students to succeed. Let move the date and make sure it is right.”

Some times the best things you ever do are things you decide not to do. Here’s to good choices.

They say if you go to a conference and come back with one good idea that you’ve had a success. Well, I had the chance to hear Dell’s CIO, Steve Schuckenbrock, speak, and I got at least one good idea from him. He had come to Dell from PepsiCo, and before that from EDS. So I asked how he went about beginning in a new organization, figuring out how to make an impact and to create change while also learning how to work within a new culture. His answer was great. He said that he starts with the assumption that everyone he meets is smart and innocent. He says if you don’t give people the benefit of the doubt at the beginning you create a self-fulfilling prophesy that people won’t be able to adapt to the need for change and to excel.

It is the first Saturday in September and my channel surfing is filled with college football games. Fall is in the air? Well, autumn is creeping in on cox digital cable, but it is 107 degrees at 3pm and my ASU football tickets look as if they will go unused tonight. (I should have offered them to the citizens of the district office building, but wouldn’t you each have said “no thanks, not that crazy.”

It’s September and we have the first two weeks of the formal semester under our belts. Our main teaching and learning environment did not fare well. First it was slow, then it was down, then certain parts of it were still slow, and during this writing it seems repaired. The root problems are mostly poorly designed software that only reveals itself when being pounded by a very large institution like Maricopa. On one hand, it’s disappointing the bad code exists. On the other hand, the vendor has done an admirable job of helping us locate the problems and writing fixes.

I could fill up this blog with comments, frustrations, suggestions and metaphorical analogies that we’ve received the past two weeks. Being a songwriter, the metaphors show some real insights and creativity. Our situation has been drawn as a parallel to B52 bombers too heavy for take off, 18 wheeler semi-trucks that could be replaced with 12 pickup trucks, and the inevitable suggestion “put a fresh hamster in the wheel.”

My own predominant analogy has been with a firefighting situation. Once the software began to malfunction under load, we had an urgent situation to deal with. When there is a fire in the canyon, the calls to figure out how the fire started, and whether we should have allowed campers in the forest, and whether we needed a controlled burn last winter — well, they are important conversations to have, but first we need to put out the fire.

Our vendor is still working on some particularly complicated fixes, but it is looking like we have the fire controlled. The next big test comes Tuesday when “high winds” are expected. The high winds will come in the form of tens of thousands of students returning to their studies after a 3 day weekend. It’s a little strange to be home on a Saturday of a three day weekend and thinking “I can’t wait for Tuesday to get here.” Stay tuned…

In early August I attended a CQIN conference in San Antonio, and heard some outstanding speakers from Toyota, Goodwill Industries, Datatel, and Valero Oil. One of the things that stuck with me was about creating organizational change. Some one mentioned that organizations do evolve and change, but that it is almost always a slower process than what the leader has in mind. It’s like a VW that can drive down the road pretty smoothly at 45, but if you take it up to 60, well you get some unwelcome feedback. It made me think about ITS. I think there is a hypothetical rate of progress that would be very smooth… but we would get passed on the highway. So, we need to be willing to move out of the comfort zone, and we have been doing that. Sometimes we can feel the vibrations. There is understandable worrying out loud about the move to Emerald Point, or the past disappointment of not moving to the content management system, and the very valid feedback that with the work demands this summer there wasn’t enough communication or empathy. There is also a very real spirit that ITS today is not what it was in 2003, and for the better.

I think it is a matter of looking for a middle ground. All things considered, we aren’t going to go 100 mph, so we have to be reasonable. On the other hand, unless we embrace some significant change we don’t reach our goals.

A little over three weeks ago I was surprised with the sudden death of my dad. I hope I have already reached everyone in expressing my gratitude for the sympathy and support that was showed to me and my family. It was a true comfort and I will always remember the reassurance of knowing that the Maricopa community was being supportive and keeping a thought for me. My “lesson learned” is to never take for granted the benefit of being part of a community and being surrounded by good-hearted people and also to never underestimate the usefulness of expressing sympathy in whatever form comes naturally. I guess that’s two lessons learned.
Not the kind of information that reaches the newspapers, but we have something good going on here.

On Wednesday, we had a power outage in the Maricopa Data Center. Here is what I sent out the next morning:

Dear Maricopa,

In case you have not already been informed through other sources, yesterday the District Office computer room suffered a power outage at approximately 12:30pm due to a malfunction between the Uninteruptible Power Supply and the Power Distribution Unit. Ironically, these devices are used to ensure a proper response to power fluctuations, but in this case they were the source of the problem, not the solution. Due to some complications in resetting the UPS and PDU, the power was out to the entire data center for approximately 3 hours.

Once power was restored, ITS went through a careful process of bringing services back online, and ensuring proper operation of the applications. This process takes about 2 hours of clock time. Almost all services were completely restored by 6pm. We had some lingering issues today caused by the after effects of computers and databases suddenly losing power. To my knowledge, these are now fixed.

As is always the case, we will take away lessons from this experience. We will make changes to the ways the UPS and PDU can be powered on and off. We will sharpen our communication, because we learned yesterday that when both phone and email are completely out, we need to have cell phone numbers, and alternate email accounts on standby. There will be other changes required as well.

We also can use this as a message that Maricopa needs a second data center. Gratefully, we had already identified and acquired a property for this use, and will begin to make plans and allocate budget to make it a reality in the months ahead.

It probably goes without saying, but we regret the outage of yesterday and we in ITS pledge to use it as a learning experience to do a better job in the future.

***********************************************************************

Now it is a couple of days later, and I have these further comments:

1. I am really proud of the core service providers in ITS. The DBA’s, the helpdesk, the server team(s), the network team, email team and the applications folks. Presented with an unexpected wrinkle, they handled it with professionalism. At one level, it was an unscheduled test of “what happens if someone pulls the plug?” We found out some things, and we had people back in business PDQ.

2. We learned that having the phones down is a bad combination with any other outage. I’ve been exploring external services that would send SMS messages to a pre-determined group. So far the one that looks promising for me to use personally is Communicator from Clickatell.com. Stay tuned for a progress report.

3. It confirms a belief I have had for a long time. The problem you prepare for is not the problem that is going to happen. So now we will prepare for having the phones out at the same time e-mail is out. However, the outage gremlins have something else in store for us, I bet.

4. As I said in my note, I am glad we have begun the process to bring up a second data center. One thing that we don’t talk about that much, because it sounds like excuse making, is that it costs money to never fail. In the months ahead we will no doubt be talking more about investing money to reduce our outages. However, I’ll predict that when we look at the total cost of doing all the things that make sure we never have an outage, we will surely decide to live with some risk so that we can keep our investments at a frugal level. Stated simply: Google and Amazon can afford some strategies that we can’t.

5. The biggest lesson learned is that work is more enjoyable with technology tools. Once the data center went down, time dragged a bit for most employees. We had things to do, but the stuff we needed to do the most required technology. In a way, that’s reassuring. We are on the right track.

We are seeing some early indications that things are coming together nicely. The Time and Labor project is complete and 9000 employees are using the web to report time instead of circulating paperwork. That saves trees and serves taxpayers better with more accurate reporting too. And since that work is in place, it lays the foundation for online handling of other payroll processes. It all comes together. We’ll be celebrating that team on June 10th.

Another glimmer occured when the Dean of IT at Rio had the following cool suggestion. I am paraphrasing: “since I have a license to sell software to authenticated Rio employees for home use, and since we have District-Wide authentication working for the Sharepoint servers to support the Blue Ribbon Panel… what if I can extend our license so that all Maricopa employees can authenticate and buy authorized software for at-home computers?” I am not filling in all the details here, but it is an example of how a couple of pieces fit together and then you can do neat things.

Another example is acquiring a building block for Blackboard to allow text messaging to students from within Blackboard. It turns out the company that does that will also enable Colleges to send text messages, essentially for the price we’ve already paid, outside of the Blackboard environment as well. Technology isn’t always complicated, sometimes it makes things simpler.
That’s why we are going to all the pain of implementing a modern Enterprise Application System for Students. It sets the foundation to be able to analyze the data in new ways, to offer more services, for more hours in the day, and then throughout 2008 and beyond people will say “Since we are already doing X, and we know how to do Y in the new system, what if we started doing Z?” And because of what the teams did in the prior years, the answer will be “Yeah, Cool. Why not?!?”

Sometimes when a team is on a winning streak, the media may tell us that they won 22 out of 28 games in June. And so, instead of July 1st, the team just says it is June 31st, June 32nd, and so on. That’s a good idea for ITS because we have had a really good month. The SIS project promised training databases to be available on June 18th. It took heroics, but it happened on time. That meant that colleges that prepared to start their own training could get started. It had been a long wait, but significant numbers of Maricopans can see the real software and plan how to serve students. Nice.

The SIS project also had a milestone to begin integration testing in June. More heroics, and that happened too. 60 people came to integration testing kickoff. That shows that across Maricopa people are mobilizing to shake the problems out during the summer so that things go smoother in October. Inspiring. We discovered that our planning for the number of remote printers was off, almost by a factor of ten. At first glance, that’s a bummer — but on second thought it is wonderful… we know NOW that we have to do some scrambling, instead of finding out on a Monday morning in mid-October. Good job.

Besides that, District Office staff located a good site for a 2nd data center, and moved with record speed to acquire it. A longstanding need is on the threshold of being achieved. Also our Identify Management development accomplished their target to roll out product by end of June. We have a place on the ITS website that summarizes all our points of pride. You can get ongoing details there. But I wanted to mention here that July has been cancelled, and June has 61 days this year.

Congrats to ITS and to the SIS/NSS project teams. Keep the winning streak going!

Today I met my younger daughter and only granddaughter at the Arizona Mills Food Court to have a SUBWAY veggie sandwich and to (maybe) ride on the Merry Go Round. This was a first for the 3 of us together. We had our meal, and watched the large mechanical beasts from afar. Once 2 year old Ashlyn got the concept that people were getting to take rides, she led us all over and picked out her horse (yellow is her favorite color and thank goodness there was exactly one yellow horse). Lindy didn’t trust the “horsebelt” and wisely stood next to her little one, so I got astride a pretty nice looking zebra that was right next door. During the 20 laps we took I noticed several people gawking and laughing at the 50 year old man with black wingtip shoes, button-down shirt, and tie, riding the carousel on his lunch hour. Wooooo hoooooooo.