Well it has been a few weeks since I have updated you. Not a lot going on that would be of much interest but I will tell you what I have been doing none the least. Since we last spoke, the project has reached a Milestone of 50%. We also reached an unbelievable Safety record of 50 Million, yes I said million man hours without a lost time accident. Woo hoo is that impressive or what? Back in the earlier days of construction, companies would factor in how many deaths they thought would happen and how it would affect the project. These days the drive is to not have any deaths, any major accidents, small accidents etc. So needless to say we are not only the safest project in the company, we are also close to a record as well.
For three weeks we have had some major I.T. or should I say IPS (Information Process Systems) issues. First we were running along just fine and then one day out of no where my file server crashed. Everything else was running but it decided it was tired I guess and decided to shut itself down. When I checked, it was in a state of disarray it had corrupted one of its system files and was unable to boot back up. I worked with the group in Dubai from 5 am that morning until about 9:30 that night. We tried everything we possibly could but to no avail. It looked like we were going to have to rebuild it from scratch which possibly we could lose data. I was shutting other servers down and swapping out CD drives because the one in the file server decided it was tired as well. We tried three different drives and none of them worked. While the team in Dubai worked with Dell to resolve our pickle, I found an old drive from a Laptop and started rebuilding the old one. By the time Dell had finished putting in their two cents, we tried one more time. This time the server booted to the CD and we were able to reload the Operating System, for you novices, that is Windows Server 2003. It is just like Windows but has features built in that allow adding security, routing and other stuff.
Next in our dilemma was that we had been working on trying to free up some space on one of the drives. I had been moving data off to another and as quickly as I moved it someone would put something else on. To also add to our burdens, we have an application server which was also running out of disk space plus needed to be upgraded to a new version. Since we had tried everything possible to fix that solution and nothing worked, we opted to install some additional drives and increase the space. The Dubai team was working to build a new server which we would use to upgrade the old one and then swing that one down to our warehouse when we were completed. Well chance would have it that the new server showed up and we would use it to replace the file server we just repaired. So we made plans to have an outage that Friday and it would encompass a lot of work. We would have to move servers, copy data, uninstall one server, install the new one, then verify that everything would work exactly as the other.
During the next couple of days, we had also planned at lunch to upgrade our Satellite modem software as well. I received all the instruction from the provider and worked on getting things ready. The process was not supposed take more than 30 or 45 minutes so we were comfortable we could get it all done and back online before lunch was over, NOT! As I prepared everything in the server room waiting on the engineers call, I realized that we had no power close to the cabinet so I scrambled to find an outlet. Unfortunately everything was taken so I had to modify an adapter to work. The engineer called and we began the process of taking everything off line and loading the first patch, well it didn’t work. We tried everything possible that either of us could think but we just could not get the patch to load. Finally after an hour and a half we abandoned the process and put everything back the way it was. While I tidied up the engineer said he would get with the manufacturer to see what the problems was. Once I got back in my office, I received a not so very nice email from the Director that said never plan an outage during the day again during working hours. Ok so I was trying to save some money and not have to work over but if that was what he wanted, so be it.
Well as Friday approached, we began moving the data from the old server to the new. Everything got transferred and it was decided that Dubai would make once last update on Friday morning and I could come in around 10 to start the swap. Day to day stuff went on and we had hiccups here and there but no major issues. Thursday came and we made final plans and preparations, Ardie would come in around 9 or so and help with the move and I readied the server room and planned the days’ events. Well the day started pretty good I slept in and grabbed a nice breakfast and just chilled a little. Around 9 Dubai called and said they were almost through copying data so I headed in. Around 9:30 or so, they were ready so we started shutting down servers and moving equipment. We battled the first server getting it out because our tools are limited. It took us about 45 minutes to get the thing out and then we started moving the servers up in the rack to make room for the beast we were installing. After fighting with that for another 45 minutes or so we decided that it could just sit on top and we would have a nice bug hole in the middle. Well this thing weighed around 150 pounds so I was trying to figure out the two of us would be able to do it. Fortunately for us there were folks in the office and borrowed a couple of additional hands. By the time we finally got the thing in the rack, we had 4 people lifting, holding and screwing the bolts back in, and a couple of non technical supervisors ;).
All in all it went pretty smoothly just very bulky and heavy and the guys holding it were under some pressure but they were troopers all the way through and never complained. Once all the cables were back in I called Dubai and told them we were coming up. I had also scheduled for that day the modem upgrade since the engineer was able to find a fix. So Dubai changed all the settings and told us to do what we needed to and they would log back in afterward and finish up. Well back to the modem once again getting everything hooked up and ready. The engineer walked me through all again and this time it worked. After he finished up we tested everything to make sure it worked, he tweaked a few things and bid hi a due. So I called Dubai back and told them we were all finished and they logged back in and went on with their synchronization. The day had been a success and I was able to get out of there by about 2:30. Still had time to chill out in the room, watch a little TIVO and catch an early dinner. So that’s just what I did. The day ended pretty quietly no burps in the new server, so I thought.
Next day back in the office, we started noticing that files we had accessed before were inaccessible, printers that existed before had disappeared just all around not what we had expected. So I got on the phone with Dubai and told them what was happening and they began to work on the issues. By the end of the day they had gotten most of them finished and everyone seemed to have access to what they needed with just a few minor printing issues. So the next morning we continued on and were able to get everything resolved by lunch.
Well I guess that is enough for now but we aren’t finished, not by a long shot. More excitement is yet to come, so stay tuned………
Specials
2 months ago
Posts
