Recently, we were working with a customer that had new diesel generators installed at several sites.  We happened to meet the generator maintenance technician at one of the sites while doing other work. An interesting discussion ensued.  It turns out that the technician, in his instructions had been told to set the maintenance routine up so that the generator ran for 15 minutes once a week, under no load.

Generators are no different than an automobile when it comes to engine maintenance.  It’s common knowledge that the worst thing you can do to an automobile engine is to only run it for 15 or 20 minutes and then shut it off. This has to do with several items that affect an internal combustion engine.

First and foremost is the oil.  Oil oxidizes while sitting in the crankcase.  This causes acids to form and chemistry to change.  What removes these acids and normalizes the oil is heat and pressure over a minimum period of time.  15 minutes is not long enough to heat the oil and maintain that heat and pressure to remove the acids that form from the oil oxidation.

Oil and heat has a secondary affect on an engine.  Oil attaches and migrates into the pores of the metal. This limits rust and other contaminates from bonding to the metals of the engine.  The classic example of this is in the form of a cast iron skillet.  Anyone who cooks knows that a well seasoned cast iron skillet will not have food stick to it. The vegetable oil permeates the pores in the iron, thus not allowing things to stick. To maintain this, you must use the skillet regularly, get it hot and allow the cooking oils to penetrate into the iron. A skillet like this also will not rust, and will last for years.

Secondly, moisture affects an engine – particularly the exhaust system.  The engine needs to heat the exhaust system enough to boil all the water condensation out.  Have you ever noticed that during the winter your car exhaust has “smoke” coming out for some time after you start it, but it disappears once the engine gets hot? That’s the water vapor escaping as the system heats up.  A generator is no different, and the exhaust system is critical to proper operation of any gen-set.

It’s important to remember that the engine is only half of the generator.  The other half, the actual generator has its issues too.  It is also affected by moisture and heat.  That is why it’s very important to exercise the system under a load.  This allows the electrical portion of the generator also to come up to operating temperature.

Lastly, the NFPA has specific standards for testing.  The critical section is in NFPA 2010 Section 110, Chapter 8.  The maintenance routine is in Section 8.4. Specifically, Section 8.4.2:

“Diesel Generator sets in services shall be exercised at least once monthly for a minimum of 30 minutes, using one of the following methods:

(2)    Under operating temperature conditions and at not less than 30 percent of the EPS nameplate kW rating”

I will take issue here: I don’t believe that monthly testing is adequate.  In our industry, and for critical sites, weekly is required, with all the above criteria met.

There are several other standards dealing with standby generators, their installation and maintenance. Specifically, NFPA 70/NEC 701, which deals directly with “Critical Operations Power Systems” and the Joint Commission on Accreditation of Healthcare Organizations. I would recommend that if you have not reviewed these standards, that you do so.

I’ve been reading about this now for about 2 weeks, and frankly, I wanted to have more to talk about.  In this link to the National Weather Service, you can read the official announcement, and here is a nice article in the Wall Street Journal. Having lived in the St. Louis area for over 35 years, I’ve seen lots of floods.  Of course, the flooding in 1993 was the worst (so far).

So, why am I posting this? A couple of reasons;

First, I know that not everybody pays attention to the weather service, or the news for that matter, and I wanted to “spread the word” in my own way.

Second, I wanted to reinforce maintenance and systems preparedness.

Some of the things that I have come across in my experience that happens during flooding and chronic wet conditions can really create headaches. You really don’t want to be troubleshooting a communications issue while trying to dispatch sand trucks or trying to maintain a switching station.

Things to look for primarily are weatherproofing issues.  Some may be beyond your control.  Water creeps into telephone cables all the time. This can cause noise and cross talk in analog circuits, and can cause errors on T1 lines. While fiber is somewhat immune to the water, the electronics are not. Additionally, if the fiber handholds and electronics are under water, the circuits will fail.

Water also does interesting things to microwave paths. Water has a high reflection coefficient. This is why we design in diversity when paths travel over large bodies of water. These ties back to overall system design.

So, my question is, when was the last time you tested your hot standby systems? Have you recently busied out your T1 and accepted calls on your backup trunks? For that matter, when was the last time you tested your UPS on some of your smaller sites or non-generator systems?  Have you forced a ring direction switch on your fiber or ring microwave system?  If you are running a simulcast system, how does this affect the timing and the system retraining time? Are your people trained and experienced enough to notice and respond to equipment alarms appropriately?

For some of my customers, these items are standard operating procedure, but for some, the network is taken for granted.  Let’s not take things for granted.

When to check your nuts

OK, so a pithy headline can get anybody’s attention, but when it comes to tower maintenance, it really is about nuts and bolts.

Towers are like bridges – they all need maintenance, but unless they break or are visible, they tend to get neglected. Oh, sure, the big ones, the ones with lights and alarms get some attention, but when was the last time you checked the ones that don’t get painted or lit? Would you be worried if the nuts at the base of your tower were like the ones in this video??

OK – so, not fair; this was a new tower. BUT we discovered this during our final punch list.  This tower was lighted, and had all antennas installed.  In fact all the civil was completed.  We only needed to install electronics at the site.

However, this photo

is not of a new tower, but one that is up.  Notice the concrete? Over time, everything fatigues, even concrete and steel.  The forces of heating and cooling naturally work parts loose, and if not checked regularly, could lead to a major catastrophe. The crack in the concrete was found while adding antennas to this tower.  The crack leads right to one of the anchor bolts.  This is a serious repair.

As part of this post, I went out to Rohn, Sabre, and Valmont to try and find the recommended maintenance intervals and practice.  Unfortunately, my search of their respective web sites was not fruitful. I’m sure an email or phone call to them with the model number of your installation would get you quick results.

The point here is to make sure that you include tower inspections and routine maintenance in your budget.  I know things are tight from a fiscal perspective, but like your communications system, a minor failure can multiply, and over time become a real issue.  And we all know that it’s during the critical times that systems tend to break down.

I saw something today that I’ve seen so many times before that I really just don’t understand.  I walked into a tower site ran by a local Public Safety organization. The site itself was, from an external standpoint fairly nice. The tower, building and guys were well fenced, and access was excellent.  However, when we entered the small transmitter building, what we found was not quite what one would expect.  Expecting nice, albeit small equipment racks with equipment neatly stacked and arranged in such a manner as to display a professional and well thought out system, we found instead, 2 AC powered console base stations, and one 12 volt powered base station.  The console base stations were in cabinets and locked, a plus in my book, the other transmitter since it was substantially newer and 12 volt powered was sitting on a plywood shelf with the cheesy metal shelf brackets screwed into the wall.


Now don’t get me wrong.  I understand that sometimes economy trumps everything else. Especially when working with public dollars.  There always seems to be too much project at the end of the budget. The console base stations however, were not bolted to the floor. The grounding, coax and other infrastructure was neatly and professionally done. The 12 volt transmitter, even though on the shelf that it is, was neatly installed. Unfortunately, the location of this site is in a known earthquake zone.  One good shake, and as the rhyme goes, they all go tumbling down.


No, what really crumbles my cookies was what these AC base stations were plugged into. One would expect, at a minimum, a properly grounded commercial line interactive UPS, complete with alarming.  A quick note; I am NOT a DC power snob. While I do like DC power in certain applications, I do realize that AC power backup in the right situation(s) is appropriate, cost effective and reliable. But, when I walk into a Public Safety radio system site, I do not expect to see critical communications gear plugged into $150 UPS’s that someone picked up from the local office supply chain. The servers here at Praecom World Headquarters have better UPS on them than these radios.


I know the folks that installed and maintained the radios.  Again, they are fine competent radio people. But radio people are not power people. Public safety people are not radio OR power people. Radio dealers are in the business to sell radios, and in today’s competitive world, anywhere they can cut costs they will.


I’m not disparaging the UPS manufacturer either. These are fine units for an office environment or supporting a small single server.  I utilize a model very similar on the telephone system here in the office. The point I’m getting to is that it was the wrong application for the design.


You see, you have several dynamics all coming together that, if the moon and the stars are in proper alignment, something is going to go wrong. The site did have generator power, so hold time generally would not be a problem. The issue truly is that there lacks any visibility into the system. If the UPS were to fail, for any reason, AND the generator failed to start, all the poor dispatcher would know is that they were off the air. And these UPS’s do fail. Regularly, and when you least expect it.


As a comparison, also in this building is where the local RBOC has brought in a SONET terminal to drop out the circuits for the radios. The SONET terminal was in a small 19” rack, bolted to the floor. In a tray in the bottom of the rack was 4 nice 25 AH gel cell batteries and a float charger. The RBOC does have deeper pockets, but they also understand reliability and power. They have learned their lessons over 100 years of needing to provide service.


So the final advice is, whether you’re spending $250,000, $50,000 or $10,000 on a site take a little extra and design the power appropriately. A small commercial UPS and generator transfer panel is, in the grand scheme of things a small cost. And I guarantee you’ll sleep better at night. At a minimum, make sure you know what you’re asking for, and in the end, what you’re getting.