Category Archives: Industrial Automation

Bob’s Clues to Doomed Project Managers

Leave a reply

I was thinking back today about a conversation I had on a plane about ten years ago. It was my first long distance “onsite” as a control systems guy, and I was very excited. Next to me on the plane to Hertfordshire, Connecticut was a grizzled old technician, about a month away from retirement. I can’t for the life of me remember his name, but the name “Bob” would suit him, so I’ll call him that.

We got to talking about what brought both of us together in that plane that day. I was off to startup some machine with a program that was “almost done”, and Bob was coming back from a service call on some CNC machines.

I spent most of my time on that flight listening to his lifetime of stories. I distinctly remember his right hand was missing the little finger and the tip of the ring finger. “A press did that to me,” he said. “Didn’t get my hand out of the way in time.” I figured it probably happened years ago, before all the new safety standards were in place. “No,” he laughed, “just four years ago. Make sure you keep your hands in your pockets!” He pointed at me with his right hand as he said it, just for emphasis.

What really got Bob talking was when I told him I wanted to be a project manager some day. “Why the hell would you want to do that?” he said. “You like white hard hats and clipboards, do ya,” he smirked. I got the sense Bob had seen a lot of project managers come and go in his day. You might say he viewed them with a critical eye. Still, we talked a bit more, and Bob related to me a kind of “system” he had for spotting, what he called, the “Doomed Project Manager”. He said all you have to do is listen for some key phrases, and that’s enough to tell you if a PM is a rising star or on the next train to dooms-ville.

Now I’ve been very fortuitous in my career. I’ve never actually heard anyone I’ve worked with ever say anything like this, and that’s certainly a testament to the professionalism and dedication to excellence of all the teams I’ve ever had the pleasure of working with. Certainly, I’ve never said any of these things. Still, I think Bob’s advice might ring true for other teams that might be struggling with their own instance of a doomed project manager. Therefore, here are Bob’s Clues to Doomed Project Managers, as closely as I can remember them:

1. “We just need to buckle down”

“If I had a nickle for every time a PM talked about buckling down,” said Bob. “What this really means is he doesn’t see any obstacle that can’t be overcome with lots of unpaid overtime. I imagine old King Tut buildin’ his pyramid talked to his people a lot about buckling down.”

“The problem,” said Bob, “is that they get addicted to it. The solution to every problem on every project is to buckle down. They don’t learn to plan it better next time, have the parts ordered a week earlier just in case, have the drawings reviewed by someone else, or whatever. Why bother when you can make it someone else’s problem?”

2. “We’re a little behind, but I think we can make it up”

“On the surface, this sounds a lot like the buckle down thing, but there’s a subtle difference. In fact you’ll often hear them used together, like we’re a little behind schedule, but I think we can make it up if we just buckle down. What you’ve really got on your hands here is an incurable optimist. The best predictor of future performance is past performance, but this fella thinks yeah, it’s been going bad so far, but for no reason whatsoever, I think it’s about to turn around and go the other direction.

3. “We know you like to do things the right way, but…”

“I’ve been around a few years now, and I find I’m now the guy they call in to go fix stuff that’s already behind schedule and over budget. So I really think it’s great when they pull you into their office and say we know you like to take the time to do things right, but we’re way behind schedule and over budget here, so take whatever shortcuts you can to get it to work, and we’ll have to fix the rest on the next job. The irony of this is apparently lost on the poor doomed PM, because it’s almost always the rushed, half-assed job that he told the guy before you to do that got him into this mess in the first place.”

4. “We didn’t have time to…”

“This is my favorite,” said Bob. “The customer comes to you and says the widget doesn’t do what he wanted it to do, and you say of course we can do that!… but let’s take a look at the spec to see if it’s supposed to be included, or if it’s an extra. You call the PM and he says, I remember them saying in a meeting they didn’t want that, but we were over budget and behind schedule so we didn’t have time to update the spec. Then you end up doing days of rework to make up for the 30 minutes of typing he saved. The PM’s whole job is to plan, document and manage the scope of the job. If he wasn’t doing that, who was managing the project?”

Others?

There were some other clues I just can’t remember, but those were the ones that stuck with me. Does anyone have clues of their own they would like added to the list?

Software Standards Run Amok

1 Reply

A few years ago I wrote a program for a customer using a flowcharting language.  It wasn’t just a flowcharting language; the product allows you to use both relay ladder logic and flowcharts.  But the customer had a software standard that made the use of ladder logic forbidden!

Imagine a simplified example.  Do you think Bill Gates’ kitchen has a blender?  Probably.  But it’s probably not like the one you or I have.  Maybe it’s made out of solid gold and maybe it has a nuclear power source, but most of all I imagine it has a fully fledged industrial control system.  Now, imagine we’re programming the control system for this blender.

Of course it would have the standard start and stop buttons, but this blender is top of the line and absolutely safe.  It has a sensor to determine if the lid is closed.  Obviously, if we’re happily blending away and the lid flies off, we need to stop the blender, and it shouldn’t start again until the lid is back on and the start button is pressed.  After all, we wouldn’t want to endanger Bill’s fingers or anything (I happen to know he does a lot of blending).

Many of you have already written this complicated control system in your mind:

ladder-logic-for-blender

Good job!  No matter what happens, the lid will have to be closed or the motor simply won’t run.  Of course, the software specification says that you can’t use ladder logic.

Well, in flowcharting the start and stop logic is simple:

 flow-chart-for-blender

That’s pretty simple, but it doesn’t take the lid closed switch into account.  At first glance, we might have to put the check for the lid closed before every decision block, so we check for the state of the lid switch, then check for the start or stop buttons.  Do you see how this could get out of hand quickly?

Fortunately, the language offers a solution:

flow-chart-for-blender-with-exception

How’s that?  While it’s true that if the lid ever comes off, the motor will stop, we have no way to exit gracefully.  This “exception block”, as it’s called, stops whatever you were doing, turns off the motor and starts the whole process over again.  I’m sure it would work fine for our simplified example here, but what if we were doing something else later on in the flowchart?  What if we were tracking parts, or decelerating an unrelated axis?  In the ladder logic example, the lid switch only disables the motor.  In the flowchart it stops the logic and then disables the motor.

Of course, to deal with this problem in the complexities of a real machine, you end up writing two different flowcharts: one for the sequence and one for the outputs.

flow-chart-for-blender-sequence-outputs

Now take a close look at the right hand flowchart.  It scans through both decisions on every single scan of the controller.  It’s “ladder logic written sideways”.  It’s simple combinational logic that screams to be rewritten in ladder.

In fact, writing the state logic (like auto mode sequences) in flowchart and the combinational logic (like manual mode and faults) in ladder makes a lot of sense, especially for more complicated machines.  So why forbid ladder logic?  Perhaps it was just to force people to start using the flowcharts.

A software standard that bans ladder logic is a bad idea.  Some logic is more readable as flowcharts, and some is more readable in ladder.  For years we’ve had to write everything in ladder.  We were like the little boy who only had a hammer and thought every problem was a nail.  If that boy suddenly traded the hammer for a screwdriver, was he any better off?

Dogs, Cats and Moody Machines

Leave a reply

Have you ever wondered what the ladder logic for a dog would look like?  I imagine it’s something like this:

ladder-logic-for-dog

… and so on.  I think that’s why we consider dogs so loyal.  Perhaps by loyal, we mean predictable, or understandable.

Ever wonder what the ladder logic for a cat would look like?  I imagine it thus:

ladder-logic-for-cat

… or something like that.  Actually I’m pretty sure I’ve met a couple of cats that came equipped with thirteen sequencers and a conditional subroutine jump in there somewhere.  It certainly makes life interesting.  Does that little tail wag mean it’s safe to pet, or does it mean your cat is about to mistake your inner thigh for prey?  Who knows!  What fun!

I guess my point is, cats can be moody, and believe it or not, so can machines.  You might call it “internal machine state”, but I call it moodiness.  Have you ever been trying to troubleshoot a machine and it was stuck thinking there was a part in one of the stations that really wasn’t there?  Every time it indexes it keeps faulting?  That’s machine moodiness.  So there you are, flagging every sensor in sight trying to get that part present bit to clear, and no matter how many roses or chocolates you buy for the darned thing, you know you’ll be sleeping in the dog house tonight.

Thankfully, there’s a cure for machine moodiness:  Make all internal state visible and editable.  At the very least, there should be a screen on the HMI that shows the current status of the part present bits at each station.  If you really want to be fancy, make sure it allows the operator to set and clear those bits manually.  That includes latches, sealed coils, counters, FIFOs, and even long running timers.

Anyway, if you can’t see it, you can’t troubleshoot it, so adding visibility will save you time in the future.  Trust me.  And trust dogs; they’re quite loyal.

Standards for the Sake of Standards

Leave a reply

I love standards.  I wish we had more of them.  I wish the IEC 61131-3 programming language standard was actually a standard and not a suggestion.  But of course, sometimes we end up with standards we could do without…

They’re a lot like those dumb laws you hear about in your email inbox like, “You cannot chain your alligator to a fire hydrant.”  You know this law only exists because someone, at some point, chained their alligator to a fire hydrant.  Some of these dumb standards exist because they are no longer relevant.  Dumb standards keep hanging around because we are so concerned about telling people what to do that we forget to say why.  Maybe this is because it seems obvious at the time. 

I was contemplating this the other day when a co-worker related to me the fable of the five monkeys (reprinted here for your convenience):

There was an interesting experiment that started with five monkeys in a cage. A banana hung inside the cage with a set of steps placed underneath it. After a while, a monkey went to the steps and started to climb towards the banana, but when he touched the steps, he set off a spray that soaked all the other monkeys with cold water. Another monkey tried to reach the banana with the same result. It didn’t take long for the monkeys to learn that the best way to stay dry was to prevent any monkey from attempting to reach the banana.

The next stage of the experiment was to remove the spray from the cage and to replace one of the monkeys with a new one. Of course, the new monkey saw the banana and went over to climb the steps. To his horror, the other monkeys attacked him. After another attempt, he learnt that if he touched the steps, he would be assaulted.

Next, another of the original five was replaced with a new monkey. The newcomer went to the steps and was attacked. The previous newcomer joined in the attack with enthusiasm!

Then, a third monkey was replaced with a new one and then a fourth. Every time a newcomer approached the steps, he was attacked. Most of the monkeys beating him had no idea why they were not allowed to climb the steps or why they were joining in the beating of the newest monkey.

After replacing the fifth monkey, none of the monkeys had ever been sprayed with water. Still, no monkey ever approached the steps. Why not? Because as far as they knew it was the way it had always been done around here… and that is how company policy begins.

Another co-worker of mine would say that this situation exists because the monkeys are only passing on data, not knowledge.  The monkeys have created a culture that is immutable to change because the culture rewards following and enforcing the rules more than understanding why the rules exist.

We need standards for efficiency.  Part of their value is as a mechanism for passing information between people.  We don’t want to re-invent the wheel and we don’t want to repeat our mistakes, but we pass up a valuable opportunity to pass on knowledge if we don’t document the why of each standard.  Imagine being a new employee and being handed the company’s electrical controls standards document.  Here’s an excerpt:

Standard 10.3(a) sub. 5: All wires will be terminated with ferrules.

I can imagine why this standard might exist – too many hours spent troubleshooting electrical problems caused by loose wire strands shorting out on nearby terminals.  But if you’re a brand new employee straight out of school, would you understand that’s why this standard exists?  When people don’t know why, they tend to make up their own reasons.

What happens two decades down the road when we’re all using carbon nanotube wires or some other non-stranded alternative?  Without knowing why the standard exists, we might try to enforce this standard in a way that doesn’t make sense.  Without a clear why stated, we risk allowing this standard to become another dumb standard, making the company less efficient.

So please, let’s make a new standard for future standards.  Every time you write a standard, include a short paragraph with it describing the history of the decision and the reasoning behind it.  Take the opportunity to pass on your knowledge!

Steps without Sequencers

2 Replies

“Why do you engineers insist on using sequencers in your PLC programs?”

Mike asked that question the last time I was over at the local washing machine manufacturer.  Mike works in the maintenance department there.  As usual, a five minute job had turned into over two hours of frustration, and I just happened to be the sympathetic shoulder.

First, let me disclose a little known secret about how washing machines work.  You see, early in the days of washing machines, certain garments, usually socks, ended up getting wrapped around the spindle causing a jam, and this eventually causes the motor to overheat and burn out.  Frustrated by this problem for over a decade, manufacturers finally solved this problem by incorporating a device known affectionately in the industry as a “sock shredder”.  When wayward socks start to get tangled around the spindle, the extra torque on the motor activates a torsion bar that engages a spinning cutting head that pulverizes the sock into fine enough fibers that most of it simply gets carried away in the rinse cycle.  Even though companies were worried about the backlash from customers, the initial consumer trials showed that most people were rather ambivalent about the missing socks, so the device is now incorporated into every washing machine on the market today.

(Not only did the laundry industry benefit from this innovation, but sock manufacturers reported a 62% rise in demand over a 5 year period, unprecedented in the undergarments industry both before and since.)

Anyway, back to Mike and his frustration.  Apparently the plant had received complaints that a few washing machines were experiencing premature failure of the motor.  Apparently the machines had left the plant without the sock shredders installed.  Mike explained to his boss that it would be easy to install a photoeye that checked for the presence of the sock shredder at the spindle insert station, and stop the inserter if the sock shredder wasn’t detected.

After wiring in the photoeye, Mike went online with the PLC, started looking for the output that actuated the inserter cylinder but was surprised to discover it wasn’t referenced as a coil.  As you can probably guess by now, the outputs were being driven by a sequencer instruction.

Ancient History

PLCs first appeared in the 60’s.  While most historians believe that PLCs were intelligently designed, a few skeptics maintain that they evolved from less advanced technologies.  But I digress…

Back in prehistoric times (before PLCs), machines used mechanical drum sequencers (just like the valve timing in older automobiles) to sequence the actions of the machine.  The drum triggered electrical cam switches that would turn on outputs in the correct sequence to operate the machine.  When PLCs first hit the market, the PLC designers thought it would be a good idea to create a “sequencer” instruction that would mimic the operation of the drum sequencer to make it easy for machine manufacturers to convert their existing machines over to PLCs.  That was decades ago.  So, why are engineers still using sequencers?

Machine builders use sequencers because they’re building a machine with a standard operation and it’s easy to go straight from a timing diagram to a sequencer.  (Some programmers will tell you that sequencers use less PLC memory, but with the price of memory these days, that’s not a valid excuse.)  Of course, when the machine hits the plant floor and needs to be modified, the presence of sequencer instructions frequently triggers severe hair loss in the maintenance department.

Solution

Sequencers can easily be replaced with logic made up of contacts and coils.  Each step can be represented by a single coil that is sealed in when the step is complete.  I find it convenient to also use a second coil for each step called “step in progress”, which is equivalent to “previous step complete and current step not complete”.

Use the “step in progress” coils to drive the outputs directly.  The logic goes from obfuscated to obvious.

Steps don’t have to be numbered either.  With tag based PLC processors you can create steps called Lather, Rinse, and Repeat.  Then if you need to add a new step called Scrub between Lather and Rinse, there’s no awkward shuffling of step numbers.

Bottom Line

For Mike’s sake, don’t use sequencers!

References


Tag Based Controllers

6 Replies

A PLC Tag is not a Description; nor is it a Variable Name

We’ve had tag based control systems for years now but some programmers just aren’t getting it.

The other day I was working on this machine that makes hand made quilts.  It’s quite an ingenious machine, actually.  The whole secret to this thing was this “Hand-O-Matic” stitcher unit that randomly inserts longer or shorter stitches to simulate making the quilt by hand.  That was the problem, though – the production manager said once in a while it just wouldn’t produce short or long stitches so the machine produces these perfectly stitched quilts that won’t even fetch a tenth of the price at the big box retailer.  The quality department kept rejecting them.

The drawings showed it was pretty simple.  The PLC has two outputs to the stitcher: one tells it to do a long stitch, the other tells it to do a short stitch.  Mike from the maintenance department had already determined that when the machine was acting up, those outputs weren’t working.

We’ve Got Tags, Let’s Use Them

I went online with the PLC and did a quick search for the output name shown in the drawings – nothing.  That’s odd.  Then I bring up the tag list and look for any tags with Stitcher in the name – nothing.  Finally I go to the hardware tag address for the outputs themselves and trace those back through the logic, and find they are being driven by some linear feedback shift registers.  Guess what the tag names for the outputs of the shift registers were…  “bits(3).5” and “bits(4).5”.  Well That’s Intuitive!

That’s an example of the first mistake I keep seeing: programmers who don’t even take advantage of tag based addresses.  This isn’t 1990 anymore; it’s time to move on.

Tags are not Variables

The second mistake is usually made by guys like me who programmed computers in school before they started programming PLCs.  The first time I saw tags I thought, “Great!  Variables!”  In languages like C, BASIC, and Pascal, the programmer has to use variable names that function as both a name and description.  You’ll frequently see variable names like “miFileLineLoopCount” – meaning, of course, that it’s a module scoped integer that counts up once every time you read another line from the file.  The only thing that keeps things sane is that they use lots of locally scoped variables that only have meaning within a single block of code, so you only have to worry about 20 or so variables at once.  In the PLC world, we’re largely stuck with tags scoped at the global or program level (though that’s changing).

So guys like me started creating PLC tags like “Inbound_Conveyor_Zone_1_Blocked”.  But the tag names are so long that when you try to view a decent sized rung of logic with 15 or so contacts, the programming software either wraps the rung in on itself or makes you scroll to the right 5 screenfuls.  It becomes unreadable.  Tags are not variables – let’s keep the descriptions separate.

Using Structured Tag Names

Every PLC program needs a structured tag naming convention.  When the machine was designed it was already broken down into zones, areas, stations, components – whatever.  Tag names should reflect this.  Have the tags make use of device IDs used in the drawings too.  It’s also important to make tag names short.  Consider this structured tag name: “S110_Deliv_PP”.  The description of that tag might be “Station 110, Delivery Conveyor, Part Present Latch [E1522PRX]”.  The beauty of ladder logic is that the description will be displayed over the contact or coil in the editor.  You don’t have to spell out “Station110…” in the tag name because anyone familiar with the system will recognize “S110…” means the same thing.

The advantage of this scheme is that the tag database can be displayed in alphabetical order, so all tags for station 110 will be grouped together, and all tags for the station 110 delivery conveyor will also be grouped together.  Do you need to add an identical delivery conveyor to station 120?  Just select all the “S110_Deliv_*” tags and copy and rename them to “S120_Deliv_*” and you know you’ve got them all.  Do you want to delete station 110?  Just select all the “S110_*” tags and delete them.  Before tag based systems this meant we had to hunt down all the addresses used for that conveyor and delete the descriptions for each one (or we could leave the descriptions in there just to confuse the next programmer who has to look at it).

So, let’s use tags with a purpose.  Come up with a structured tag naming convention and stick to it.

Oh… and if anyone’s interested in buying some authentic hand made quilts, let me know – I can get you a great deal!