Author Archives: Scott Whitlock

About Scott Whitlock

I'm Scott Whitlock, an "automation enthusiast". By day I'm a PLC and .NET programmer at ETBO Tool & Die Inc., a manufacturer.

ZigBee… Where are you?

2 Replies

Years ago I experimented with X10 in our old house. It was cheap and fun, but it was unreliable. Not long after, I heard about this great new technology called ZigBee that would form a wireless mesh network between all the devices in your house, and communications between modules would retry and reroute around problems. I was impressed! It seemed to good to be true!

Apparently it was. Here we are, 5 years later, and where is my X10 replacement? From what I can tell, the electric utilities are starting to deploy ZigBee (and our house in Ontario was just upgraded with a “smart meter” which most likely has ZigBee technology in it). Great, but I can’t seem to find a single off-the-shelf ZigBee home automation product. It’s been 5 years!

Is it time to forget about ZigBee for Home Automation? We’ve blanketed our houses in WiFi. It can’t be that hard to put an 802.11 chip in a wall socket. It looks like Control4‘s products do just that. Anyone know of any off-the-shelf WiFi enabled light switches?

Automating the Home, Failed

2 Replies

A couple of years ago we moved into a nice new home, and like most new homes it had a shiny new HRV.  I had never seen one before, and I was impressed by the idea that it exchanged stale inside air with fresh outside air but it conserves the energy you’ve already invested in heating or cooling the inside of your home.  I actually pulled the cover off to study how it worked, and pulled the cover off the furnace to see how it was hooked in.

The HRV draws air from the bathrooms, kitchen and laundry areas in the house (because they are your sources of moisture) and exhausts that air outside, but brings in an equal amount of air.  Rather than distributing the incoming air back into the house itself, the HRV dumps it into the furnace airflow, and sends a signal to the furnace to run whenever the HRV is on.

When we received the keys to our home, the builder instructed us to leave the HRV on at its lowest level 24 hours a day to prevent humidity, particularly at the beginning as the home is wet from construction and you need to let it dry out.  Makes sense.  After about a year though, I had this idea that running the furnace fan 24/7 at full speed just because the HRV was running at its lowest speed probably wasn’t the most energy efficient setup, particularly since the HRV itself is supposed to be an energy saving device.  After doing a bit of research, I discovered the manufacturer of our HRV made something called a Lifestyles Controller.  This is basically just like a programmable thermostat, but it controls your HRV.  You can program it to start and stop at different times of the day, operate in different modes and at different speeds depending on the need, etc.  Great!

HRV-ControllerMy odyssey started when I attempted to purchase this marvel of technology.  The manufacturer wouldn’t sell me one because I needed to go through an authorized distributor, and of course there was only one in my area.  I contacted them and they’d never heard of this device.  I gave them the part number and they had to get back to me.  When they called me back, they asked me why I wanted to buy it, and acted like this was a huge inconvenience for them (imagine, a customer calls you and wants to buy something).  That should have been my first clue of the dangers to come.  I insisted that I wanted to buy one, and yes, I’m quite capable of installing it myself, and yes, I understand it’s a special order item and there are no returns.  Did I mention it was expensive too?

When I went to pick it up at the distributor, they were very careful to tell me that I must unplug the HRV before installing this or else I could easily short out the power supply.  I don’t mean they told me this once or put a note on the package to remind me, but several people made it really clear not to do anything with it plugged in.  Makes sense.  I’m used to rewiring stuff and I always turn the power off.  Again, this was a sign, and I didn’t heed it.

I got the device home, unpacked it and set to work installing it making sure to unplug the HRV first.  The instructions were very simple and easy to understand.  I didn’t bother running wires through walls at this point; I just wired it up about 3 feet away from the HRV (my history in automation has taught me to avoid commitment).  When I finished wiring, I plugged the HRV back in, the lifestyles controller powered up, and… wouldn’t control the HRV.  I followed the instructions to the letter and just couldn’t get it to ever start the HRV even with the button that just manually starts the HRV on a timer.  I tried a couple of other ideas, like disconnecting the existing bathroom control pads, etc. (unplugging it every time!), but nothing seemed to work.  I dreaded the next obvious step.

I called the distributor.  “The lifestyles controller for my HRV doesn’t work.”

“Did you install it without unplugging the HRV?”

You can imagine the rest of that conversation.  Nothing surprising.  They asked me to call the manufacturer and talk to their tech support.  Again, “Did you install it without unplugging the HRV?”  Grrr.

The manufacturer told me I had to take it back to the distributor who would return it to them, which was quite funny since all three of us were located in the same city!  Anyway, here I am back in the distributor’s office, and the woman behind the counter takes a look at the device, bewildered, looks at my receipt and says, “This is a special order item.  There are no returns.”

“I don’t want to return it.  I just want one that works.”

“Oh.”

They did send it back and lo and behold, there was nothing wrong with the device.

“Wait, what?!?”

“They tested it and there’s nothing wrong with it.  However, it’s only compatible with HRV firmwares after October 2006.  When did you purchase your HRV?”

“We got the house in September of 2006.”  At least that made sense.  Maybe we were getting somewhere.

“The manufacturer has offered to provide an upgraded controller board free of charge, but it’s not something you can install yourself.  We’ll have to come and install it, and our hourly rate is blah and there’s a minimum charge of blah.”  (I can’t remember the exact price, and I’d probably just get angry if I did.)

At that point I figured, if I’m paying a minimum charge anyway, why not have them come and install the whole thing, and get it put in the right spot, run the wires, etc.  Great.  Wait, it gets better.

Buddy comes to install the gadget.  He’s got the new controller board, and gets that installed pretty quickly, but when he asks me where I want the controller, I tell him I want it right beside the existing programmable thermostat, in the hall upstairs.  He takes a walk around, takes a look from the basement, turns to me and tells me he didn’t bring a ladder with him.  No problem, I say, I have a couple in the garage, and it’s not terribly surprising that he wouldn’t have a ladder anyway.  Wait, it gets better.

He also had to borrow my drill, and an electrical fish to get the wire through the wall.  Wait, it gets better.

“What gauge wire do you figure this needs?” he says.

You’re asking me?  I have to take a day off work to pay you hundreds of dollars to borrow my tools to install a product you sell and you’re asking me what gauge of wire you need?  Wait… it gets better.

“I didn’t bring any wire with me.”

“No problem,” I say.  “I’ve got a whole whack of telephone cable in the garage, and it’s at least the gauge of wire that the rest of the HRV communication uses, so it should be fine.”

So we set to work running the wire.  Just to put the icing on the cake, when we go to punch the hole in the drywall where the controller is going to be mounted, we discover there’s a plywood blocker just behind the drywall in that spot.  I grab the drill, put a spade bit in it, and drill an inch diameter hole through the plywood.  Then I discover there’s another wire back there… interesting.  I just drilled through it actually… I wonder what that belongs to.  It turns out that was the wire that goes to the glass break detector on the security system.  Had to fix that too, but that was my fault.

We got the whole thing installed and wow!  It worked!  Buddy gave me a discount for having to use my tools, and I was a happy geek, programming my HRV.

That is, until a few months later when I noticed the humidity in the house was through the roof.  The controller was still “on”, but it was obvious it had stopped controlling the HRV.  I’m not sure what caused it to fail, but thankfully it was still under warranty.  I got it replaced.  The one I got back works.  I’m holding my breath…

That’s my latest foray into home automation, and I’m not impressed.  In my job in industrial automation, I can actually install and program the industrial controllers myself, and when I talk to the distributors and manufacturers, nobody treats me like a moron.  Home automation is still in its infancy, so the only people doing this themselves are generally early adopters like myself: technical and geeky.  Treating us like this is completely counter-productive.  I hope some company, somewhere, steps up to the plate and starts working with us, not against us.

Software Standards Run Amok

1 Reply

A few years ago I wrote a program for a customer using a flowcharting language.  It wasn’t just a flowcharting language; the product allows you to use both relay ladder logic and flowcharts.  But the customer had a software standard that made the use of ladder logic forbidden!

Imagine a simplified example.  Do you think Bill Gates’ kitchen has a blender?  Probably.  But it’s probably not like the one you or I have.  Maybe it’s made out of solid gold and maybe it has a nuclear power source, but most of all I imagine it has a fully fledged industrial control system.  Now, imagine we’re programming the control system for this blender.

Of course it would have the standard start and stop buttons, but this blender is top of the line and absolutely safe.  It has a sensor to determine if the lid is closed.  Obviously, if we’re happily blending away and the lid flies off, we need to stop the blender, and it shouldn’t start again until the lid is back on and the start button is pressed.  After all, we wouldn’t want to endanger Bill’s fingers or anything (I happen to know he does a lot of blending).

Many of you have already written this complicated control system in your mind:

ladder-logic-for-blender

Good job!  No matter what happens, the lid will have to be closed or the motor simply won’t run.  Of course, the software specification says that you can’t use ladder logic.

Well, in flowcharting the start and stop logic is simple:

 flow-chart-for-blender

That’s pretty simple, but it doesn’t take the lid closed switch into account.  At first glance, we might have to put the check for the lid closed before every decision block, so we check for the state of the lid switch, then check for the start or stop buttons.  Do you see how this could get out of hand quickly?

Fortunately, the language offers a solution:

flow-chart-for-blender-with-exception

How’s that?  While it’s true that if the lid ever comes off, the motor will stop, we have no way to exit gracefully.  This “exception block”, as it’s called, stops whatever you were doing, turns off the motor and starts the whole process over again.  I’m sure it would work fine for our simplified example here, but what if we were doing something else later on in the flowchart?  What if we were tracking parts, or decelerating an unrelated axis?  In the ladder logic example, the lid switch only disables the motor.  In the flowchart it stops the logic and then disables the motor.

Of course, to deal with this problem in the complexities of a real machine, you end up writing two different flowcharts: one for the sequence and one for the outputs.

flow-chart-for-blender-sequence-outputs

Now take a close look at the right hand flowchart.  It scans through both decisions on every single scan of the controller.  It’s “ladder logic written sideways”.  It’s simple combinational logic that screams to be rewritten in ladder.

In fact, writing the state logic (like auto mode sequences) in flowchart and the combinational logic (like manual mode and faults) in ladder makes a lot of sense, especially for more complicated machines.  So why forbid ladder logic?  Perhaps it was just to force people to start using the flowcharts.

A software standard that bans ladder logic is a bad idea.  Some logic is more readable as flowcharts, and some is more readable in ladder.  For years we’ve had to write everything in ladder.  We were like the little boy who only had a hammer and thought every problem was a nail.  If that boy suddenly traded the hammer for a screwdriver, was he any better off?

Dogs, Cats and Moody Machines

Leave a reply

Have you ever wondered what the ladder logic for a dog would look like?  I imagine it’s something like this:

ladder-logic-for-dog

… and so on.  I think that’s why we consider dogs so loyal.  Perhaps by loyal, we mean predictable, or understandable.

Ever wonder what the ladder logic for a cat would look like?  I imagine it thus:

ladder-logic-for-cat

… or something like that.  Actually I’m pretty sure I’ve met a couple of cats that came equipped with thirteen sequencers and a conditional subroutine jump in there somewhere.  It certainly makes life interesting.  Does that little tail wag mean it’s safe to pet, or does it mean your cat is about to mistake your inner thigh for prey?  Who knows!  What fun!

I guess my point is, cats can be moody, and believe it or not, so can machines.  You might call it “internal machine state”, but I call it moodiness.  Have you ever been trying to troubleshoot a machine and it was stuck thinking there was a part in one of the stations that really wasn’t there?  Every time it indexes it keeps faulting?  That’s machine moodiness.  So there you are, flagging every sensor in sight trying to get that part present bit to clear, and no matter how many roses or chocolates you buy for the darned thing, you know you’ll be sleeping in the dog house tonight.

Thankfully, there’s a cure for machine moodiness:  Make all internal state visible and editable.  At the very least, there should be a screen on the HMI that shows the current status of the part present bits at each station.  If you really want to be fancy, make sure it allows the operator to set and clear those bits manually.  That includes latches, sealed coils, counters, FIFOs, and even long running timers.

Anyway, if you can’t see it, you can’t troubleshoot it, so adding visibility will save you time in the future.  Trust me.  And trust dogs; they’re quite loyal.

Standards for the Sake of Standards

Leave a reply

I love standards.  I wish we had more of them.  I wish the IEC 61131-3 programming language standard was actually a standard and not a suggestion.  But of course, sometimes we end up with standards we could do without…

They’re a lot like those dumb laws you hear about in your email inbox like, “You cannot chain your alligator to a fire hydrant.”  You know this law only exists because someone, at some point, chained their alligator to a fire hydrant.  Some of these dumb standards exist because they are no longer relevant.  Dumb standards keep hanging around because we are so concerned about telling people what to do that we forget to say why.  Maybe this is because it seems obvious at the time. 

I was contemplating this the other day when a co-worker related to me the fable of the five monkeys (reprinted here for your convenience):

There was an interesting experiment that started with five monkeys in a cage. A banana hung inside the cage with a set of steps placed underneath it. After a while, a monkey went to the steps and started to climb towards the banana, but when he touched the steps, he set off a spray that soaked all the other monkeys with cold water. Another monkey tried to reach the banana with the same result. It didn’t take long for the monkeys to learn that the best way to stay dry was to prevent any monkey from attempting to reach the banana.

The next stage of the experiment was to remove the spray from the cage and to replace one of the monkeys with a new one. Of course, the new monkey saw the banana and went over to climb the steps. To his horror, the other monkeys attacked him. After another attempt, he learnt that if he touched the steps, he would be assaulted.

Next, another of the original five was replaced with a new monkey. The newcomer went to the steps and was attacked. The previous newcomer joined in the attack with enthusiasm!

Then, a third monkey was replaced with a new one and then a fourth. Every time a newcomer approached the steps, he was attacked. Most of the monkeys beating him had no idea why they were not allowed to climb the steps or why they were joining in the beating of the newest monkey.

After replacing the fifth monkey, none of the monkeys had ever been sprayed with water. Still, no monkey ever approached the steps. Why not? Because as far as they knew it was the way it had always been done around here… and that is how company policy begins.

Another co-worker of mine would say that this situation exists because the monkeys are only passing on data, not knowledge.  The monkeys have created a culture that is immutable to change because the culture rewards following and enforcing the rules more than understanding why the rules exist.

We need standards for efficiency.  Part of their value is as a mechanism for passing information between people.  We don’t want to re-invent the wheel and we don’t want to repeat our mistakes, but we pass up a valuable opportunity to pass on knowledge if we don’t document the why of each standard.  Imagine being a new employee and being handed the company’s electrical controls standards document.  Here’s an excerpt:

Standard 10.3(a) sub. 5: All wires will be terminated with ferrules.

I can imagine why this standard might exist – too many hours spent troubleshooting electrical problems caused by loose wire strands shorting out on nearby terminals.  But if you’re a brand new employee straight out of school, would you understand that’s why this standard exists?  When people don’t know why, they tend to make up their own reasons.

What happens two decades down the road when we’re all using carbon nanotube wires or some other non-stranded alternative?  Without knowing why the standard exists, we might try to enforce this standard in a way that doesn’t make sense.  Without a clear why stated, we risk allowing this standard to become another dumb standard, making the company less efficient.

So please, let’s make a new standard for future standards.  Every time you write a standard, include a short paragraph with it describing the history of the decision and the reasoning behind it.  Take the opportunity to pass on your knowledge!

Steps without Sequencers

2 Replies

“Why do you engineers insist on using sequencers in your PLC programs?”

Mike asked that question the last time I was over at the local washing machine manufacturer.  Mike works in the maintenance department there.  As usual, a five minute job had turned into over two hours of frustration, and I just happened to be the sympathetic shoulder.

First, let me disclose a little known secret about how washing machines work.  You see, early in the days of washing machines, certain garments, usually socks, ended up getting wrapped around the spindle causing a jam, and this eventually causes the motor to overheat and burn out.  Frustrated by this problem for over a decade, manufacturers finally solved this problem by incorporating a device known affectionately in the industry as a “sock shredder”.  When wayward socks start to get tangled around the spindle, the extra torque on the motor activates a torsion bar that engages a spinning cutting head that pulverizes the sock into fine enough fibers that most of it simply gets carried away in the rinse cycle.  Even though companies were worried about the backlash from customers, the initial consumer trials showed that most people were rather ambivalent about the missing socks, so the device is now incorporated into every washing machine on the market today.

(Not only did the laundry industry benefit from this innovation, but sock manufacturers reported a 62% rise in demand over a 5 year period, unprecedented in the undergarments industry both before and since.)

Anyway, back to Mike and his frustration.  Apparently the plant had received complaints that a few washing machines were experiencing premature failure of the motor.  Apparently the machines had left the plant without the sock shredders installed.  Mike explained to his boss that it would be easy to install a photoeye that checked for the presence of the sock shredder at the spindle insert station, and stop the inserter if the sock shredder wasn’t detected.

After wiring in the photoeye, Mike went online with the PLC, started looking for the output that actuated the inserter cylinder but was surprised to discover it wasn’t referenced as a coil.  As you can probably guess by now, the outputs were being driven by a sequencer instruction.

Ancient History

PLCs first appeared in the 60’s.  While most historians believe that PLCs were intelligently designed, a few skeptics maintain that they evolved from less advanced technologies.  But I digress…

Back in prehistoric times (before PLCs), machines used mechanical drum sequencers (just like the valve timing in older automobiles) to sequence the actions of the machine.  The drum triggered electrical cam switches that would turn on outputs in the correct sequence to operate the machine.  When PLCs first hit the market, the PLC designers thought it would be a good idea to create a “sequencer” instruction that would mimic the operation of the drum sequencer to make it easy for machine manufacturers to convert their existing machines over to PLCs.  That was decades ago.  So, why are engineers still using sequencers?

Machine builders use sequencers because they’re building a machine with a standard operation and it’s easy to go straight from a timing diagram to a sequencer.  (Some programmers will tell you that sequencers use less PLC memory, but with the price of memory these days, that’s not a valid excuse.)  Of course, when the machine hits the plant floor and needs to be modified, the presence of sequencer instructions frequently triggers severe hair loss in the maintenance department.

Solution

Sequencers can easily be replaced with logic made up of contacts and coils.  Each step can be represented by a single coil that is sealed in when the step is complete.  I find it convenient to also use a second coil for each step called “step in progress”, which is equivalent to “previous step complete and current step not complete”.

Use the “step in progress” coils to drive the outputs directly.  The logic goes from obfuscated to obvious.

Steps don’t have to be numbered either.  With tag based PLC processors you can create steps called Lather, Rinse, and Repeat.  Then if you need to add a new step called Scrub between Lather and Rinse, there’s no awkward shuffling of step numbers.

Bottom Line

For Mike’s sake, don’t use sequencers!

References


Tag Based Controllers

6 Replies

A PLC Tag is not a Description; nor is it a Variable Name

We’ve had tag based control systems for years now but some programmers just aren’t getting it.

The other day I was working on this machine that makes hand made quilts.  It’s quite an ingenious machine, actually.  The whole secret to this thing was this “Hand-O-Matic” stitcher unit that randomly inserts longer or shorter stitches to simulate making the quilt by hand.  That was the problem, though – the production manager said once in a while it just wouldn’t produce short or long stitches so the machine produces these perfectly stitched quilts that won’t even fetch a tenth of the price at the big box retailer.  The quality department kept rejecting them.

The drawings showed it was pretty simple.  The PLC has two outputs to the stitcher: one tells it to do a long stitch, the other tells it to do a short stitch.  Mike from the maintenance department had already determined that when the machine was acting up, those outputs weren’t working.

We’ve Got Tags, Let’s Use Them

I went online with the PLC and did a quick search for the output name shown in the drawings – nothing.  That’s odd.  Then I bring up the tag list and look for any tags with Stitcher in the name – nothing.  Finally I go to the hardware tag address for the outputs themselves and trace those back through the logic, and find they are being driven by some linear feedback shift registers.  Guess what the tag names for the outputs of the shift registers were…  “bits(3).5” and “bits(4).5”.  Well That’s Intuitive!

That’s an example of the first mistake I keep seeing: programmers who don’t even take advantage of tag based addresses.  This isn’t 1990 anymore; it’s time to move on.

Tags are not Variables

The second mistake is usually made by guys like me who programmed computers in school before they started programming PLCs.  The first time I saw tags I thought, “Great!  Variables!”  In languages like C, BASIC, and Pascal, the programmer has to use variable names that function as both a name and description.  You’ll frequently see variable names like “miFileLineLoopCount” – meaning, of course, that it’s a module scoped integer that counts up once every time you read another line from the file.  The only thing that keeps things sane is that they use lots of locally scoped variables that only have meaning within a single block of code, so you only have to worry about 20 or so variables at once.  In the PLC world, we’re largely stuck with tags scoped at the global or program level (though that’s changing).

So guys like me started creating PLC tags like “Inbound_Conveyor_Zone_1_Blocked”.  But the tag names are so long that when you try to view a decent sized rung of logic with 15 or so contacts, the programming software either wraps the rung in on itself or makes you scroll to the right 5 screenfuls.  It becomes unreadable.  Tags are not variables – let’s keep the descriptions separate.

Using Structured Tag Names

Every PLC program needs a structured tag naming convention.  When the machine was designed it was already broken down into zones, areas, stations, components – whatever.  Tag names should reflect this.  Have the tags make use of device IDs used in the drawings too.  It’s also important to make tag names short.  Consider this structured tag name: “S110_Deliv_PP”.  The description of that tag might be “Station 110, Delivery Conveyor, Part Present Latch [E1522PRX]”.  The beauty of ladder logic is that the description will be displayed over the contact or coil in the editor.  You don’t have to spell out “Station110…” in the tag name because anyone familiar with the system will recognize “S110…” means the same thing.

The advantage of this scheme is that the tag database can be displayed in alphabetical order, so all tags for station 110 will be grouped together, and all tags for the station 110 delivery conveyor will also be grouped together.  Do you need to add an identical delivery conveyor to station 120?  Just select all the “S110_Deliv_*” tags and copy and rename them to “S120_Deliv_*” and you know you’ve got them all.  Do you want to delete station 110?  Just select all the “S110_*” tags and delete them.  Before tag based systems this meant we had to hunt down all the addresses used for that conveyor and delete the descriptions for each one (or we could leave the descriptions in there just to confuse the next programmer who has to look at it).

So, let’s use tags with a purpose.  Come up with a structured tag naming convention and stick to it.

Oh… and if anyone’s interested in buying some authentic hand made quilts, let me know – I can get you a great deal!