Category Archives: Automation

Lowering the Cost of Automation Equipment

5 Replies

(This is the third part of a trilogy of blog posts about what I think is the biggest roadblock preventing significant growth in the industrial automation industry. You should read Part 1: Industrial Automation Technology is Stagnant and Part 2: Why Automation Equipment Vendors Dabble in Integration first.)

I was recently making significant additions to a PLC program and the resulting program was too big to fit in the existing PLC. There weren’t many areas where the code could be made smaller without impacting readability, so I went out and priced the same series of PLC with more memory. I’m not going to name brands or distributors here (and I’d probably get in a lot of trouble if I did, because they don’t publish their prices), but I was taken aback by one thing: the only difference between these two controllers was the amount of memory (execution speed and features were the same), and it was over $1000 more. I realize this is “industrial” equipment, so the memory probably has increased temperature ratings, and it’s battery backed, but how much PLC memory can you buy for $1000? As of writing this post, the cost of 1 Gibabyte of memory for a desktop computer is less than $50.

At $50 per Gig, 20 Gigs of PLC memory? Nope.

Ok, it’s industrial, so… 1 Gig? Nope.

Try less than half a Megabyte. Yes.

Just for the sake of comparison, how much is that per Gigabyte? If half a Meg costs $1000, then the price of PLC memory is 2 Million dollars per Gigabyte. WTF? It’s not 40 times more expensive, it’s forty thousand times more expensive than commodity RAM.

Do you know why they charge two million dollars per gigabyte for PLC memory? Vendor lock-in. It’s that simple. They can charge thousands of dollars because the switching costs are insanely high. I can either buy a PLC in the same series from the same manufacturer (and they make sure there’s only one distributor in my area) with more memory, or I can replace the PLC with one from another manufacturer incurring the following costs:

  • Rewriting the program because the languages are not portable.
  • Modifying the electrical drawings significantly.
  • Rewiring the electrical panel.
  • Retraining all of my existing maintenance and engineering staff on the new platform.

So it really doesn’t matter. They could charge us $5000 for that upgrade and it’s still cheaper than the alternative. Why don’t they publish their prices? Simple, they price it by how much money they think you have, not by how much it costs them to produce the equipment. Wouldn’t we all like to work in an industry like that?

How did it get this way? In the PC industry, if Dell starts charging me huge markups on their equipment, I can just switch to another supplier of PC equipment. Even in the industrial PC world, many vendors offer practically equivalent industrial PCs. You can buy a 15″ panel mount touch screen PC with one to two gigs of RAM for about $3000. Add $500 and you can probably get an 80 Gig solid state hard drive in it. Compare that with the fact that small manufacturing plants are routinely paying $4500 for a 10″ touch screen HMI that barely has the processing power to run Windows CE, and caps your application file at a few dozen megabytes!

The difference between the PC platform and the PLC platform comes down to one thing: interoperability standards. The entire explosive growth of the PC industry was based on IBM creating an open standard with the first IBM PC. Here’s the landscape of personal computers before the IBM PC (pre 1980)*:

Here’s the market share shortly after introduction of the IBM PC (1980 to 1984):

… and here’s what eventually happened (post 1984):

Nearly every subsystem of a modern PC is based on a well defined standard. All motherboards and disk drives use the same power connectors, all expansion cards were based on ISA, or later PCI, and AGP standards. External devices used standard RS232, and later USB ports. Hardware that wasn’t interoperable just didn’t survive.

That first graph is exactly like where we are now in the automation industry with PLCs. None of the vendors have opened their architecture for cloning, so there’s no opportunity for the market to commoditize the products. Let’s look at this from the point of view of a PLC manufacturer for a moment. Opening up your platform to cloning seems like a really risky move. Wouldn’t all of your customers move to cheaper clones? Wouldn’t it drive down the price of your equipment? Yes, those things would happen.

But look at what else would happen:

  • The hardware cost of your platform would drop, so everyone would leave your competitor’s platforms and come to yours. At least you’d still be in the game, but your competitors wouldn’t even have a compatible platform anymore.
  • By dropping the price of automation equipment, the demand for complementary products would increase. IBM didn’t make its fortune directly in the PC industry, but they rode the wave and became an enterprise services company built on top of the commodity PC business.
  • The size of the industry would explode.

Why don’t we have open standards? What about IEC-61131-3? PLC vendors tell us that their controllers are IEC 61131-3 compliant, but that doesn’t mean they’re compatible. The standard only specifies what languages must be supported and what basic features those languages should have. Why didn’t the committee at least specify a common file format? You may be surprised to learn that many of the major committee members were actually from the PLC equipment manufacturers themselves. The same people who have a vested interest in maintaining their own vendor lock-in were the ones designing the standard. No wonder!

What about the new XML standard? Well, it’s being specified by the same group and if you read their documentation, you can clearly see that this XML standard is not meant for porting programs between PLC brands, but rather to integrate the PLC programming tools with 3rd party programs like debugging, visualization, and simulation tools. You can be certain that a common vendor-neutral file format will not be the result of this venture.

Solutions

I see three ways this problem could be solved:

  1. An association of common manufacturers and system integrators could form and develop a truly open and interoperable standard,
  2. A PLC manufacturer could “go IBM” and open up their platform, or
  3. A single integrator or manufacturer could develop and create an open automation platform and put the rights in the public domain.

Option 1 is almost doomed to fail because it has the word committee in it, but options 2 and 3 (which are similar) have happened before in other industries and stand a good chance of success here.

In my opinion, the most likely candidate to fulfill option 2 is Beckhoff. I say this because they’ve already opened up their EtherCAT fieldbus technology as an open standard, and their entire strategy is based around leveraging existing commodity hardware. Their PLC is actually a real-time runtime on a PC based system, and their EtherCAT I/O uses commodity Ethernet chipsets. All they would really need to do is open source their TwinCAT I/O and TwinCAT PLC software. Their loss of software license revenue could be balanced by increased demand for their advanced software libraries in the short term, and increased demand for their EtherCAT industrial I/O in the longer term. Since none of the other automation vendors have a good EtherCAT offering, this could launch Beckhoff into the worldwide lead relatively quickly.

For option 3, any company that considers automation equipment to be an expense or a complementary product (i.e. manufacturing plants and system integrators) could do this. There is a long term ROI here by driving down the cost of automation equipment. Many internet companies do this on a daily basis. IBM sells servers and enterprise services, which is the natural complement of operating systems, so they invest heavily in Linux development (not to drive down the cost of Linux – it’s free – but to offer a competitive alternative to Microsoft to keep Microsoft’s software costs down). Google does everything it can to increase internet usage by the public, so it invests heavily in free services like Google Maps, Gmail, and even the Firefox web browser. The more people who surf, the more people who see Google’s advertising, and the more money they make.

If I were going to go about it, I’d build it like this:

  • Base it on commodity PC hardware (x86).
  • Pick an open source, free real time operating system like FreeRTOS.
  • Document and publish (free download) the communication protocol and file format for the automation program files (e.g. ladder logic).
  • Write a reference runtime that conforms to this communication protocol and publish it under a BSD license. It doesn’t have to be great, but it has to work and be useful.
  • Write an extensible programming environment (Windows-based) where you can develop automation programs that conform to the standard and that communicates with any runtime that conforms to the standard. Give it away free and publish it under the GNU GPL license to prevent it from being embraced and extended in a proprietary way by one vendor.

If you did that, anyone could make hardware that’s compatible with this platform, people are free to innovate on the runtime, and anyone can make improvements to the programming environment (as long as they give their changes back to the community). Instant de-facto standard.

I know some of you may be grinning right now because this is the path I’m following with my next personal software project, that I’m calling “Snap” (which stands for “Snap is Not A PLC”). However, Snap is for the home automation industry, not the industrial automation industry. If any company out there wants to take up the gauntlet and do this for industrial automation equipment, you may just change this industry and solve our biggest problem.

Discussion is welcome. Please comment below or contact me by email (e8pzrhs02@sneakemail.com).

* Graphs courtesy of ars technica.

Why Automation Equipment Vendors Dabble in Integration

Leave a reply

(This is part 2 of a trilogy. You should read Part 1: Industrial Automation Technology is Stagnant first, and then you can read the thrilling conclusion: Lowering the Cost of Automation Equipment.)

Tell me if this sounds familiar… you’re a systems integrator and you’ve been asked to come and bid on a project. At the bid line-up meeting, you sit down in the meeting room, and you look across the table, and there’s someone from the equipment vendor XYZ’s integration services department.

Of course the customer has specified that you have to use equipment from vendor XYZ, and of course the integration services wing of vendor XYZ can purchase this equipment internally (at a discount) so they can obviously undercut your price. This always left a bad taste in my mouth, but thankfully some vendors have stopped doing this.

But ask yourself why they would do this in the first place. Integration is a cut-throat market and the margins in the equipment business are significantly higher than doing integration. What business sense does it make to expand into a low margin market and risk annoying the very companies who are out there pushing your product to the end customer?

“Commoditize Your Complements”

Everyone knows the economic principle of supply and demand. If supply goes up, price goes down, but if demand goes up, price goes up. That explains what happens when equipment vendors offer integration services and undercut the competition: the supply of integration services goes up, and prices go down. This is their goal. This bears repeating: Their goal is to reduce the price of integration services in general.

Let’s talk about competitive products vs. complementary products for a moment. When a competitor lowers their price, you have to lower yours, but when the price of a complementary product goes down, demand for your product goes up, and therefore the price of your product goes up. What are complementary products? They’re products that are often purchased together, like razors and razor blades, cars and gas, or even automation equipment and integration services.

Gillette made a fortune realizing that if they reduced the price of razors, the demand for blades would go up. When the price of gas goes down, people buy bigger cars. Likewise, the automation equipment vendors know that if they can reduce the price of integration services, the demand for automation equipment will go up, and they can charge more for their product.

While I appreciate the value of competition, it needs to be a fair market. By choking off integrators, the vendors are just promoting a race to the bottom of quality.

Personally, I don’t think it has to be this way. In future posts I’m going to discuss things that the integration industry can do to reverse this trend and stop the downward spiral.

Make More Money with ControlsOverload

2 Replies

Thinking back to my childhood, I find it odd why I remember some memories so vividly while most are lost to time. I have this one absolutely clear memory. I was walking home with my mother (most likely carrying a rented copy of Star Wars or Empire Strikes Back from the local video store, which was a novelty at the time). I must have been going on about something I read about robots… I think I got a book about robots around that age and I saw pictures of some of the first industrial robots of the time. Anyway, I can clearly remember my mother telling me that she was afraid everyone was going to lose their job to a robot soon.

I think the reason this memory stuck with me was how contrary it was to the culture of kids at the time. We idolized robots. The coolest things in our lives were droids, AT-ATs and Transformers (my, how things have changed, eh?). But this wasn’t over-the-top paranoia, it was the generally accepted truth at that time. For the most part, those anxieties have vanished along with fears of nuclear war, only to be replaced by new anxieties of terrorism.

The truth is that we’ve been replacing people with robots for the last quarter of a century, and the unemployment rate peaked in the early 80’s and has been going down, not up! How is it possible that unemployment has fallen while automation is rising?

A History Lesson

In 1865, William Stanley Jevons published a book called The Coal Question. “In it, Jevons observed that England’s consumption of coal soared after James Watt introduced his coal-fired steam engine, which greatly improved the efficiency of Thomas Newcomen’s earlier design … Jevons argued that any further increases in efficiency in the use of coal would tend to increase the use of coal. Hence, it would tend to increase, rather than reduce, the rate at which England’s deposits of coal were being depleted.”

This effect became known as the Jevons Paradox:

In economics, the Jevons paradox … is the proposition that technological progress that increases the efficiency with which a resource is used, tends to increase (rather than decrease) the rate of consumption of that resource. It is historically called the Jevons Paradox as it ran counter to popular intuition. However, the situation is well understood in modern economics. In addition to reducing the amount needed for a given use, improved efficiency lowers the relative cost of using a resource – which increases demand and speeds economic growth, further increasing demand.

Automation Increases the Efficiency of Human Resources

Workers are a resource too. Automation is just a technological means to improve the efficiency of people, allowing one person to do the work of many. Since there is no cap on the demand for work, automation increases the demand for workers, since each worker becomes more valuable. Hence, unemployment goes down and wages go up.

ControlsOverload Improves Your Efficiency

When I first got into controls, you learned everything by reading a paper manual (if you could find one) or by trial and error, or if you were lucky, from someone else in your company who had used this technology before. We wasted hours trying to figure stuff out, but we accepted this because it was normal. But it also made us inefficient.

Google came along but largely passed us by. Vendors keep their knowledge bases behind locked doors on the internet, and when working on the plant floor, we rarely had internet access anyway, but that’s changing. Many of us now have smartphones so we can access the internet from anywhere. Sometimes we even have WiFi.

Enter ControlsOverload, Web 2.0 meets Controls Engineering. You don’t need to pay anyone to join. You don’t even need to register to ask or answer a question. It asks that you leave your name, email address, and website, but you can make these up if you want to remain anonymous.

When you start typing your question into ControlsOverload, it automatically searches the existing questions to see if it’s already been asked. As the amount of content on the site grows, the chance that you’ll get your answer immediately grows as well. If the question hasn’t been asked before, posting your question is a single click away. When you get answers, registered users can vote them up or down. The best answers rise to the top, so you don’t have to search through 7 pages of forum posts to find the most relevant or correct answer.

Questions are also “tagged” by technology. Perhaps you’re a vendor and you want to see all the questions related to your products. Here’s the page for all Allen-Bradley technologies for instance. If you want, you can subscribe to the RSS feed for the same page.

Will it work? The technology is already a huge success story for traditional PC programming, along with other topics like parenting and personal finance. It works. People participate because it’s really useful.

The point of ControlsOverload is to make us more efficient, as an industry. We waste less time figuring out how to do the simple stuff, like how to copy symbols out of a PLC into Excel and vice-versa.

Doesn’t this mean we’re giving away our valuable knowledge for free? Are we essentially automating ourselves out of a job?

There’s no doubt our jobs will change. We will spend less time doing trial and error and more time solving customers’ problems. The customer doesn’t care about editing PLC programs; they care about making their facility more efficient. If we can solve that problem faster and easier, then we’re more efficient. If we’re more efficient, the demand for our services goes up.

Whether you participate or not, ControlsOverload is going to increase the demand for controls engineers, and that’s going to increase the wages of controls engineers. ControlsOverload will make you more money.

On the other hand, if your business strategy is to be the only one on the block who remembers the arcane technical details of a Fanuc RJ3iB controller, I’m sorry to tell you that the viability of that business model is quickly being exhausted.

Join us on ControlsOverload:

  • Pool your knowledge.
  • Make connections.
  • Be more productive.

Industrial Automation Technology is Stagnant

5 Replies

(This is part 1 of a trilogy. When you’re finished reading this, you can read Part 2: Why Automation Equipment Vendors Dabble in Integration and Part 3: Lowering the Cost of Automation Equipment.)

I’m not the first person to notice that industrial automation technology isn’t keeping up with the rapid pace of technological change that we’ve all grown used to. Jim Pinto commented on it years ago:

Industrial automation business seems to be stuck in the mold of being a slow-growth, stable business. This generates a mindset, perhaps even complacency that inhibits change. In my opinion it’s marketing myopia, an unwillingness to think “outside the box”.

Most major automation companies have gross-profit margins of 40-50%, the industry mindset. Many other large companies in other businesses generate much lower gross margins, in the region of 20-25%. Some of this may be accounting differences, but the broad-brush differences are there. Traditionally, automation business is based on higher gross margins and lower net-profit, and it’s difficult to think outside that box.

In the industrial market, only 2-3% is spent on R&D, with some lower than that. By contrast, many of high-tech manufacturers spend 10-15% – Cisco 15-20%, with HP, Lucent, in the 8-10% range, and Motorola and Ericsson at 10-15%. One wonders what the impact would be, if industrial automation companies doubled, or even tripled, their R&D investments.

How is it that an industry like industrial automation equipment can have margins in the 40 to 50% range when their primary customers are manufacturers, probably the most price-conscious market in the world today? I’ll tell you why: they’re smart. Here’s what they did…

Originally they all sold interchangeable parts. You built a control system out of power supplies, relays, terminal blocks and valve banks. All of these parts are available, in replacement form, from various manufacturers and distributors so competition keeps margins tight. Even if you only used P&B relays, it’s pretty easy to switch your standard to Phoenix if you can get a better deal.

Then they came out with something revolutionary: the PLC. Manufacturers jumped on it because you could take four double-door panels worth of relays and replace it with a single unit, and even reprogram it in real time! It was truly a marvel of technology, but it shifted the balance of power in the industry away from the customers and towards the vendors. Now this single piece of equipment that controls the assembly line isn’t interchangeable with other vendors. Not only that, but it requires expensive programming software, not to mention a steep ramp-up time for the maintenance and engineering departments to learn the technology. Manufacturers started standardizing on “automation platforms”. Obviously this creates vendor lock-in. Once you standardize on one technology, the vendors can pretty much count on your continued business as long as they provide good support and a clear upgrade path. The barrier to switching technologies is so great that vendors can afford to have high margins, and they don’t feel the pressure to innovate like in other markets.

Unfortunately this means manufacturers are losing. They’re not getting as much value for their money as they could be. Here are some examples of what a PLC should be able to do as we approach the year 2010 (but can’t):

  • Open an ODBC connection to a database and execute basic create, read, update and delete (CRUD) instructions, straight from ladder logic, without the need for expensive 3rd party software.
  • Send an email or text message when a machine goes down, or on any sort of event.
  • Support for Secure Socket Layer (SSL) the same technology your banking website uses. This would prevent exposure of some of our critical infrastructure to network security threats.

Another unfortunate side effect is reduced innovation in the system integration space. Let’s say you’re a system integrator and you want to develop some product for a niche market, you have to choose an automation technology to base it on. Once you’ve chosen that controls technology, say Allen-Bradley, you’re also locked in. You’re going to find it hard to sell your product to a customer that has standardized on Siemens, which means you have to invest extra effort, money, and time in re-engineering your product to work on their platform. Worse, their platform may not have all the features that your product requires, so it may not work as well. It’s not much of a product if you have to re-work the design every time, which means it’s harder to recoup your development cost.

Ultimately, automation technology is more expensive and less powerful because automation equipment vendors have stopped innovating.

Wearable Machine Vision in the Home

2 Replies

I recently watched the TED video about Pranav Mistry demonstrating his “Sixth Sense” technology, and my head is spinning with ideas about the future of home automation:

At first I was blown away by the potential of the Microsoft Surface technology. They keep talking about using it on a coffee table, but for some reason, I always envision my refrigerator door as a Surface, covered with digital photos from our photo album, digital post-it notes, to-do lists, etc.. Maybe even little spelling games that my daughter can play with on the bottom of it, just like she plays with magnets and letters now.

But the sixth sense technology fundamentally shifts those ideas. First of all is price. The Surface is $10,000 (that’s an expensive refrigerator, by any stretch of the imagination). Of course the price will come down, but the components in the Sixth Sense demo are off the shelf and only cost $300, plus he’s open-sourcing the software! There’s no reason you couldn’t combine the two ideas either. Just mount the Sixth Sense system statically over your coffee table or above your fridge and you’ll get an (albeit reduced performance) similar system for a fraction of the cost.

What else could Sixth Sense do in the home? Synchronize your paper calendar on the wall with your Google calendar? Automatically pause the TV when you get up to leave the room? How about when someone from a charity comes to your door asking for donations… could I hold up their information brochure in front of me and let it look up their score on Charity Navigator and project the score on the brochure for me?

The possibilities are impressive. What would you do with Sixth Sense?

Bob’s Clues to Doomed Project Managers

Leave a reply

I was thinking back today about a conversation I had on a plane about ten years ago. It was my first long distance “onsite” as a control systems guy, and I was very excited. Next to me on the plane to Hertfordshire, Connecticut was a grizzled old technician, about a month away from retirement. I can’t for the life of me remember his name, but the name “Bob” would suit him, so I’ll call him that.

We got to talking about what brought both of us together in that plane that day. I was off to startup some machine with a program that was “almost done”, and Bob was coming back from a service call on some CNC machines.

I spent most of my time on that flight listening to his lifetime of stories. I distinctly remember his right hand was missing the little finger and the tip of the ring finger. “A press did that to me,” he said. “Didn’t get my hand out of the way in time.” I figured it probably happened years ago, before all the new safety standards were in place. “No,” he laughed, “just four years ago. Make sure you keep your hands in your pockets!” He pointed at me with his right hand as he said it, just for emphasis.

What really got Bob talking was when I told him I wanted to be a project manager some day. “Why the hell would you want to do that?” he said. “You like white hard hats and clipboards, do ya,” he smirked. I got the sense Bob had seen a lot of project managers come and go in his day. You might say he viewed them with a critical eye. Still, we talked a bit more, and Bob related to me a kind of “system” he had for spotting, what he called, the “Doomed Project Manager”. He said all you have to do is listen for some key phrases, and that’s enough to tell you if a PM is a rising star or on the next train to dooms-ville.

Now I’ve been very fortuitous in my career. I’ve never actually heard anyone I’ve worked with ever say anything like this, and that’s certainly a testament to the professionalism and dedication to excellence of all the teams I’ve ever had the pleasure of working with. Certainly, I’ve never said any of these things. Still, I think Bob’s advice might ring true for other teams that might be struggling with their own instance of a doomed project manager. Therefore, here are Bob’s Clues to Doomed Project Managers, as closely as I can remember them:

1. “We just need to buckle down”

“If I had a nickle for every time a PM talked about buckling down,” said Bob. “What this really means is he doesn’t see any obstacle that can’t be overcome with lots of unpaid overtime. I imagine old King Tut buildin’ his pyramid talked to his people a lot about buckling down.”

“The problem,” said Bob, “is that they get addicted to it. The solution to every problem on every project is to buckle down. They don’t learn to plan it better next time, have the parts ordered a week earlier just in case, have the drawings reviewed by someone else, or whatever. Why bother when you can make it someone else’s problem?”

2. “We’re a little behind, but I think we can make it up”

“On the surface, this sounds a lot like the buckle down thing, but there’s a subtle difference. In fact you’ll often hear them used together, like we’re a little behind schedule, but I think we can make it up if we just buckle down. What you’ve really got on your hands here is an incurable optimist. The best predictor of future performance is past performance, but this fella thinks yeah, it’s been going bad so far, but for no reason whatsoever, I think it’s about to turn around and go the other direction.

3. “We know you like to do things the right way, but…”

“I’ve been around a few years now, and I find I’m now the guy they call in to go fix stuff that’s already behind schedule and over budget. So I really think it’s great when they pull you into their office and say we know you like to take the time to do things right, but we’re way behind schedule and over budget here, so take whatever shortcuts you can to get it to work, and we’ll have to fix the rest on the next job. The irony of this is apparently lost on the poor doomed PM, because it’s almost always the rushed, half-assed job that he told the guy before you to do that got him into this mess in the first place.”

4. “We didn’t have time to…”

“This is my favorite,” said Bob. “The customer comes to you and says the widget doesn’t do what he wanted it to do, and you say of course we can do that!… but let’s take a look at the spec to see if it’s supposed to be included, or if it’s an extra. You call the PM and he says, I remember them saying in a meeting they didn’t want that, but we were over budget and behind schedule so we didn’t have time to update the spec. Then you end up doing days of rework to make up for the 30 minutes of typing he saved. The PM’s whole job is to plan, document and manage the scope of the job. If he wasn’t doing that, who was managing the project?”

Others?

There were some other clues I just can’t remember, but those were the ones that stuck with me. Does anyone have clues of their own they would like added to the list?

ZigBee… Where are you?

2 Replies

Years ago I experimented with X10 in our old house. It was cheap and fun, but it was unreliable. Not long after, I heard about this great new technology called ZigBee that would form a wireless mesh network between all the devices in your house, and communications between modules would retry and reroute around problems. I was impressed! It seemed to good to be true!

Apparently it was. Here we are, 5 years later, and where is my X10 replacement? From what I can tell, the electric utilities are starting to deploy ZigBee (and our house in Ontario was just upgraded with a “smart meter” which most likely has ZigBee technology in it). Great, but I can’t seem to find a single off-the-shelf ZigBee home automation product. It’s been 5 years!

Is it time to forget about ZigBee for Home Automation? We’ve blanketed our houses in WiFi. It can’t be that hard to put an 802.11 chip in a wall socket. It looks like Control4‘s products do just that. Anyone know of any off-the-shelf WiFi enabled light switches?

Automating the Home, Failed

2 Replies

A couple of years ago we moved into a nice new home, and like most new homes it had a shiny new HRV.  I had never seen one before, and I was impressed by the idea that it exchanged stale inside air with fresh outside air but it conserves the energy you’ve already invested in heating or cooling the inside of your home.  I actually pulled the cover off to study how it worked, and pulled the cover off the furnace to see how it was hooked in.

The HRV draws air from the bathrooms, kitchen and laundry areas in the house (because they are your sources of moisture) and exhausts that air outside, but brings in an equal amount of air.  Rather than distributing the incoming air back into the house itself, the HRV dumps it into the furnace airflow, and sends a signal to the furnace to run whenever the HRV is on.

When we received the keys to our home, the builder instructed us to leave the HRV on at its lowest level 24 hours a day to prevent humidity, particularly at the beginning as the home is wet from construction and you need to let it dry out.  Makes sense.  After about a year though, I had this idea that running the furnace fan 24/7 at full speed just because the HRV was running at its lowest speed probably wasn’t the most energy efficient setup, particularly since the HRV itself is supposed to be an energy saving device.  After doing a bit of research, I discovered the manufacturer of our HRV made something called a Lifestyles Controller.  This is basically just like a programmable thermostat, but it controls your HRV.  You can program it to start and stop at different times of the day, operate in different modes and at different speeds depending on the need, etc.  Great!

HRV-ControllerMy odyssey started when I attempted to purchase this marvel of technology.  The manufacturer wouldn’t sell me one because I needed to go through an authorized distributor, and of course there was only one in my area.  I contacted them and they’d never heard of this device.  I gave them the part number and they had to get back to me.  When they called me back, they asked me why I wanted to buy it, and acted like this was a huge inconvenience for them (imagine, a customer calls you and wants to buy something).  That should have been my first clue of the dangers to come.  I insisted that I wanted to buy one, and yes, I’m quite capable of installing it myself, and yes, I understand it’s a special order item and there are no returns.  Did I mention it was expensive too?

When I went to pick it up at the distributor, they were very careful to tell me that I must unplug the HRV before installing this or else I could easily short out the power supply.  I don’t mean they told me this once or put a note on the package to remind me, but several people made it really clear not to do anything with it plugged in.  Makes sense.  I’m used to rewiring stuff and I always turn the power off.  Again, this was a sign, and I didn’t heed it.

I got the device home, unpacked it and set to work installing it making sure to unplug the HRV first.  The instructions were very simple and easy to understand.  I didn’t bother running wires through walls at this point; I just wired it up about 3 feet away from the HRV (my history in automation has taught me to avoid commitment).  When I finished wiring, I plugged the HRV back in, the lifestyles controller powered up, and… wouldn’t control the HRV.  I followed the instructions to the letter and just couldn’t get it to ever start the HRV even with the button that just manually starts the HRV on a timer.  I tried a couple of other ideas, like disconnecting the existing bathroom control pads, etc. (unplugging it every time!), but nothing seemed to work.  I dreaded the next obvious step.

I called the distributor.  “The lifestyles controller for my HRV doesn’t work.”

“Did you install it without unplugging the HRV?”

You can imagine the rest of that conversation.  Nothing surprising.  They asked me to call the manufacturer and talk to their tech support.  Again, “Did you install it without unplugging the HRV?”  Grrr.

The manufacturer told me I had to take it back to the distributor who would return it to them, which was quite funny since all three of us were located in the same city!  Anyway, here I am back in the distributor’s office, and the woman behind the counter takes a look at the device, bewildered, looks at my receipt and says, “This is a special order item.  There are no returns.”

“I don’t want to return it.  I just want one that works.”

“Oh.”

They did send it back and lo and behold, there was nothing wrong with the device.

“Wait, what?!?”

“They tested it and there’s nothing wrong with it.  However, it’s only compatible with HRV firmwares after October 2006.  When did you purchase your HRV?”

“We got the house in September of 2006.”  At least that made sense.  Maybe we were getting somewhere.

“The manufacturer has offered to provide an upgraded controller board free of charge, but it’s not something you can install yourself.  We’ll have to come and install it, and our hourly rate is blah and there’s a minimum charge of blah.”  (I can’t remember the exact price, and I’d probably just get angry if I did.)

At that point I figured, if I’m paying a minimum charge anyway, why not have them come and install the whole thing, and get it put in the right spot, run the wires, etc.  Great.  Wait, it gets better.

Buddy comes to install the gadget.  He’s got the new controller board, and gets that installed pretty quickly, but when he asks me where I want the controller, I tell him I want it right beside the existing programmable thermostat, in the hall upstairs.  He takes a walk around, takes a look from the basement, turns to me and tells me he didn’t bring a ladder with him.  No problem, I say, I have a couple in the garage, and it’s not terribly surprising that he wouldn’t have a ladder anyway.  Wait, it gets better.

He also had to borrow my drill, and an electrical fish to get the wire through the wall.  Wait, it gets better.

“What gauge wire do you figure this needs?” he says.

You’re asking me?  I have to take a day off work to pay you hundreds of dollars to borrow my tools to install a product you sell and you’re asking me what gauge of wire you need?  Wait… it gets better.

“I didn’t bring any wire with me.”

“No problem,” I say.  “I’ve got a whole whack of telephone cable in the garage, and it’s at least the gauge of wire that the rest of the HRV communication uses, so it should be fine.”

So we set to work running the wire.  Just to put the icing on the cake, when we go to punch the hole in the drywall where the controller is going to be mounted, we discover there’s a plywood blocker just behind the drywall in that spot.  I grab the drill, put a spade bit in it, and drill an inch diameter hole through the plywood.  Then I discover there’s another wire back there… interesting.  I just drilled through it actually… I wonder what that belongs to.  It turns out that was the wire that goes to the glass break detector on the security system.  Had to fix that too, but that was my fault.

We got the whole thing installed and wow!  It worked!  Buddy gave me a discount for having to use my tools, and I was a happy geek, programming my HRV.

That is, until a few months later when I noticed the humidity in the house was through the roof.  The controller was still “on”, but it was obvious it had stopped controlling the HRV.  I’m not sure what caused it to fail, but thankfully it was still under warranty.  I got it replaced.  The one I got back works.  I’m holding my breath…

That’s my latest foray into home automation, and I’m not impressed.  In my job in industrial automation, I can actually install and program the industrial controllers myself, and when I talk to the distributors and manufacturers, nobody treats me like a moron.  Home automation is still in its infancy, so the only people doing this themselves are generally early adopters like myself: technical and geeky.  Treating us like this is completely counter-productive.  I hope some company, somewhere, steps up to the plate and starts working with us, not against us.

Software Standards Run Amok

1 Reply

A few years ago I wrote a program for a customer using a flowcharting language.  It wasn’t just a flowcharting language; the product allows you to use both relay ladder logic and flowcharts.  But the customer had a software standard that made the use of ladder logic forbidden!

Imagine a simplified example.  Do you think Bill Gates’ kitchen has a blender?  Probably.  But it’s probably not like the one you or I have.  Maybe it’s made out of solid gold and maybe it has a nuclear power source, but most of all I imagine it has a fully fledged industrial control system.  Now, imagine we’re programming the control system for this blender.

Of course it would have the standard start and stop buttons, but this blender is top of the line and absolutely safe.  It has a sensor to determine if the lid is closed.  Obviously, if we’re happily blending away and the lid flies off, we need to stop the blender, and it shouldn’t start again until the lid is back on and the start button is pressed.  After all, we wouldn’t want to endanger Bill’s fingers or anything (I happen to know he does a lot of blending).

Many of you have already written this complicated control system in your mind:

ladder-logic-for-blender

Good job!  No matter what happens, the lid will have to be closed or the motor simply won’t run.  Of course, the software specification says that you can’t use ladder logic.

Well, in flowcharting the start and stop logic is simple:

 flow-chart-for-blender

That’s pretty simple, but it doesn’t take the lid closed switch into account.  At first glance, we might have to put the check for the lid closed before every decision block, so we check for the state of the lid switch, then check for the start or stop buttons.  Do you see how this could get out of hand quickly?

Fortunately, the language offers a solution:

flow-chart-for-blender-with-exception

How’s that?  While it’s true that if the lid ever comes off, the motor will stop, we have no way to exit gracefully.  This “exception block”, as it’s called, stops whatever you were doing, turns off the motor and starts the whole process over again.  I’m sure it would work fine for our simplified example here, but what if we were doing something else later on in the flowchart?  What if we were tracking parts, or decelerating an unrelated axis?  In the ladder logic example, the lid switch only disables the motor.  In the flowchart it stops the logic and then disables the motor.

Of course, to deal with this problem in the complexities of a real machine, you end up writing two different flowcharts: one for the sequence and one for the outputs.

flow-chart-for-blender-sequence-outputs

Now take a close look at the right hand flowchart.  It scans through both decisions on every single scan of the controller.  It’s “ladder logic written sideways”.  It’s simple combinational logic that screams to be rewritten in ladder.

In fact, writing the state logic (like auto mode sequences) in flowchart and the combinational logic (like manual mode and faults) in ladder makes a lot of sense, especially for more complicated machines.  So why forbid ladder logic?  Perhaps it was just to force people to start using the flowcharts.

A software standard that bans ladder logic is a bad idea.  Some logic is more readable as flowcharts, and some is more readable in ladder.  For years we’ve had to write everything in ladder.  We were like the little boy who only had a hammer and thought every problem was a nail.  If that boy suddenly traded the hammer for a screwdriver, was he any better off?

Dogs, Cats and Moody Machines

Leave a reply

Have you ever wondered what the ladder logic for a dog would look like?  I imagine it’s something like this:

ladder-logic-for-dog

… and so on.  I think that’s why we consider dogs so loyal.  Perhaps by loyal, we mean predictable, or understandable.

Ever wonder what the ladder logic for a cat would look like?  I imagine it thus:

ladder-logic-for-cat

… or something like that.  Actually I’m pretty sure I’ve met a couple of cats that came equipped with thirteen sequencers and a conditional subroutine jump in there somewhere.  It certainly makes life interesting.  Does that little tail wag mean it’s safe to pet, or does it mean your cat is about to mistake your inner thigh for prey?  Who knows!  What fun!

I guess my point is, cats can be moody, and believe it or not, so can machines.  You might call it “internal machine state”, but I call it moodiness.  Have you ever been trying to troubleshoot a machine and it was stuck thinking there was a part in one of the stations that really wasn’t there?  Every time it indexes it keeps faulting?  That’s machine moodiness.  So there you are, flagging every sensor in sight trying to get that part present bit to clear, and no matter how many roses or chocolates you buy for the darned thing, you know you’ll be sleeping in the dog house tonight.

Thankfully, there’s a cure for machine moodiness:  Make all internal state visible and editable.  At the very least, there should be a screen on the HMI that shows the current status of the part present bits at each station.  If you really want to be fancy, make sure it allows the operator to set and clear those bits manually.  That includes latches, sealed coils, counters, FIFOs, and even long running timers.

Anyway, if you can’t see it, you can’t troubleshoot it, so adding visibility will save you time in the future.  Trust me.  And trust dogs; they’re quite loyal.