Here’s one for you home automation and gadget enthusiasts. Can’t afford a penthouse with a view overlooking your favorite scenery? Well, you can now bring the scenery to you, courtesy of two flat screen TVs on edge, and a wiimote hack. They even have instructions on how to build one yourself!
I have to say that this newspaper article from 1992 predicted our future right on the money. I’m an “automation enthusiast”, so I’m all about the time saving device. Just for fun I entered “time saving device” into Google and clicked “I’m Feeling Lucky”. I was directed to this question:
If every year time saving devices are invented (i.e. computers, cell phones, email, etc…) then why do we still need a 40 hour work week?
As I’ve said before, this is another manifestation of the Jevons Paradox. If I wanted to hire someone to paint a building and I had to choose between someone with a bucket of paint and a paintbrush, or a guy with a power sprayer, the latter is undoubtedly more expensive per hour, but cheaper per square foot of wall. He’ll also have the job done faster. That means the demand for the guy with the sprayer is higher. His time saving device gives him more work and less time.
In Industrial Automation, we never really talk about saving time. We talk about saving labor and saving money. We talk about increased productivity, efficiency, throughput and uptime. Time is a constant.
Yet at home we talk about time like it’s a variable. As if you could have more of it. What an odd concept when you think about it…
I’ve recently spent some time trying to think of a good time saving home automation device to create, and I’m convinced it’s a pointless activity. Would an automatic pet feeder save me time? Possibly, but you still have to refill it, the parts are likely to wear out, and it has to be cleaned regularly. It takes one of us less than 30 seconds to feed the dog every day. That’s about 3 hours per year. But if you spent 10 minutes a month cleaning it, that’s 2 hours a year right there, and how long are you going to spend fiddling with it when it finally breaks down? Even if there was no extra time, would you spend $100 to save 3 hours? It’s marginal, but doubtful.
Now where that pet feeder really shines is if you have a dog with a special need. It can feed your animal up to 8 times a day, so if your dog was diabetic, then you can start to see that it solves a difficult problem really effectively.
Would a robot vacuum save me time? I think you’ll find that it fails in the same way that the pet feeder fails: you have to clean out the vacuum bucket every time it cycles, and you have to take it apart to clean the brushes regularly. The batteries need replacing, and most reviews I’ve seen indicate that you still need to vacuum once every couple of weeks. It doesn’t save you time. It gives you the ability to vacuum more often.
Look at the three successful categories of home automation: HVAC, lighting, and audio/video. None of those are about saving you time. They’re all about improving the (perceived) quality of life. A new robot that folded your laundry for you wouldn’t be as successful as a closet that color co-ordinates your outfits for you. Everyone can fold laundry but not everyone can pick an outfit. There’s a market for the latter.
I got ahold of some Phidgets on the weekend. These are basically USB I/O devices for amateur robotics, but I’m looking at their use in the home automation space.
They have drivers for lots of operating systems, and APIs for almost any programming platform under the sun.
Getting the I/O connected and controlled from a .NET application was a breeze, including the Interface Kit 8/8/8 and the little R/C Servo Controller. That little servo could certainly move a damper in a heating duct, and they have lots of environmental sensors. The wheels have started turning…
Programming both PCs and PLCs sometimes gets me thinking about programming from a higher level. I’ve written a lengthy answer over on StackOverflow about the differences between PC and PLC programming. What I haven’t talked about before is how they are the same.
First, let me define what I mean by PC and PLC programming. By PC programming, I’m generally referring to imperative programming. There are actually two popular PC programming paradigms: imperative and declarative, and the paradigm with new-found popularity, functional, is actually a subset of declarative programming.
Most PLC programming falls into the declarative category, and most PC programming falls into the imperative category. For examples:
PC:
PLC:
So normally when we talk about the differences between PC and PLC programming, we’re talking about the differences between imperative and declarative programming, but there’s obviously overlap on both sides of the fence.
The major difference, however, is audience. In North America at least, we write PC programs with the expectation that other programmers will have to read, understand and make changes to them, but we write PLC programs with the expectation that people in the maintenance department will be expected to go online with and troubleshoot our programs. Just think about how odd that would be in the PC world: when a word processor crashes, nobody whips out their debugger, figures out what caused the program to crash, makes a fix and continues writing their letter. Primarily this is because the source code doesn’t come with the word processor, but it’s also because the programming language can only be understood by programmers.
When you look at what makes a PC program good or bad, on a high level it’s the same thing that makes a PLC program good or bad: readability. Now as I’ve pointed out, the people who have to read the program is different in each case, but really readability is the fundamental measure by which experienced programmers rate programs.
On the PC side ,the name of the game with readability is modularity. You want to divide up your program into parts, and you want to make those individual parts as self-contained as possible. You want to minimize the interaction to these parts as much as possible. That makes it easier to reason about the program because you’re abstracting away the underlying complexity on each piece and leaving a less complex interface that you can interact with. The entire domain of object oriented programming is an extension of the concept of modularity.
On the PLC side, readability is equivalent to being able to troubleshoot the machine when it’s down. Experienced PLC programmers ask themselves, “if this machine stopped unexpectedly and I had to figure out why it stopped, what would I do? How can I make it easier for someone following that process to figure out what’s wrong with the machine?”
It turns out that most people troubleshooting a machine follow a similar procedure: you start at the outputs and you work your way backwards. You generally have a good idea what the machine is supposed to do next (e.g. move slide A to position B). You can look at the print set, or even the valve itself and figure out what output should be turning on. You look at the indicator on the output card and it’s not on, so the logic isn’t telling it to turn on. You crack open the laptop, and you find that output. You’re looking for one thing: the COIL.
Notice the one big mistake you could make if you’re writing a program: you could use a whole bunch of set and reset (or latch and unlatch) instructions to drive your outputs. Based on my description, you can easily see why that would make the program less readable: which set instruction is the one that’s supposed to be turning on the output right now? If there’s only one, it’s easy, but if there are 10, you’re already lost.
Let’s assume you do find the coil that drives this output. Your next step is to follow the logic back through the rungs, right clicking on the conditions that aren’t satisfied and cross referencing until you understand what the machine is waiting for. What are some obvious mistakes you can make that would hinder this process?
Once you start thinking from the point of view of someone troubleshooting the machine, your perspective on good vs. bad programming really changes. You realize that techniques that seem to save you time while you’re programming end up costing the company hours of lost production time while maintenance picks their way through your cryptic logic.
Next time you’re writing your ladder logic, think of the poor maintenance guy who has to figure out what’s wrong, and try to make his life a little less miserable.
I’ve noticed some Industrial Automation vendors on Twitter recently (not to mention some automation related magazines).
I like that trend, but here’s some commentary on who’s doing what right and wrong (from an engineer’s perspective anyway):
After my recent rant about Rockwell Automation and their frustrating online support, I received an email from Joe Harkulich, Global Quality Leader at Rockwell Automation. The first thing he did was, helpfully, get me straightened out with my TechConnect ID. Apparently the one we were given wasn’t our real TechConnect ID, and it happened to be the ID of another company in Mexico, which caused some confusion on my end. Once that got straightened out, I could access the article I was looking for (a bit late, but better than never).
Joe then invited me to join a conference call with Esther Beris (Rockwell Knowledgebase), Jon Furniss (TechConnect), and Rob Snyder (Senior Manager for Rockwell’s remote support). I’m thankful for them taking the time to do this, as I’m really only a moderate user of Rockwell Automation products. (I really don’t think they’re very concerned about my legions of blog readers and twitter followers). I’ll relate a few of the details of the call here, and I really want to express my gratitude to everyone involved for their time.
The first item on the agenda was Joe addressing some misinformation in my previous blog post. I had made a comment about Beckhoff’s 350MB knowledge-base available for download, and I had suggested that Rockwell should do something similar. It turns out that Rockwell does offer TechConnect subscribers the ability to receive 3 DVD’s full of all their product manuals and their entire knowledge base. Joe offered to send me a copy, and I accepted. I received them yesterday and installed them on the laptop at work. This will be nice pending an on-site trip that’s scheduled next month. I’m a bit concerned that the discs are dated May of 2008. I was expecting them to be about 6 months old. We did discuss the idea of making them automatically download updates of the knowledge base, etc., so your offline copy is always up to date, but they pointed out that this would be a huge amount of information so it probably couldn’t happen, but they would take the idea into consideration. Personally I think a solution like that would have a lot of value.
We then turned our discussion towards some of the issues I had raised with Rockwell’s online support in my previous post. I certainly made it clear that I have always been impressed with Rockwell’s paid telephone support, and I really have nothing to complain about there; they are simply awesome! When it comes to the online support, we talked about these issues:
It makes no sense to me why you have to sign on separately at www.rockwellautomation.com and at their knowledgebase. Apparently it doesn’t make a lot of sense to the folks at Rockwell either, and they’re working on it. However, there’s no time-line for the resolution of it (or at least they couldn’t give me one). I think it’s one of those cases where the politics of a big organization are getting in the way of doing something really simple, and I can appreciate that. I did get the impression that the people involved do care, and it will get done, eventually.
In past years I would frequently do a lot of automation work, and then switch to PC programming for a project or two, and then switch back to automation work as I’ve recently done. I always found it a bit frustrating the first time I tried to download some EDS file and Rockwell had canceled my account because it had been longer than 6 months since I’d signed in. Apparently I’m not the only customer to complain about this, and Rockwell is in the process of changing this. Bravo!
Let’s say I go to the ControlLogix Product Page. On a normal website when I look at a product I see exactly what that one component looks like, a link directly to the datasheet or user manual, a tab with all the specs on it, and all the information I need to buy that product. But the ControlLogix page is a brand page, not a product page. It’s almost completely useless to me as an Engineer. There is a Literature link, and that has a selection guide and something else, but then it tells me I have to go to the literature library to find all the good documentation.
When I explained this on the conference call I got some good laughs, because again, I’m not the first person to complain about this. Apparently this is being resolved, and they do have a time-line: September 2010. That’s great news and I’m really happy to hear it.
I’m afraid the issue of paid vs. unpaid content is still a sticking point for me. They explained to me that they chose a paid model for their support because they can provide better support. There’s no question in my mind that the quality of Rockwell’s telephone support comes through time and again, but I have to disagree with the pay-wall model they’re using for online support. I really think what’s happened is that the technical support group within Rockwell is literally scared of what could happen if they didn’t maintain a strangle-hold over the information that guys like me need to do their job.
I’ve talked before about vendor lock-in in the industrial automation industry, so I’m not going to go into that again. But this is a case where a company is selling me upwards of $10,000 of automation equipment per project and once they’ve done that, they want to charge me extra for access to the information I need to make it work the way it was supposed to work in the first place. I understand the paid telephone support model because there’s a one-to-one relationship between the amount of time I spend on the phone and the amount of time they have to pay someone to talk on the phone. But while they’re doing that, they’re already creating a knowledge-base of information. As smarter people than I have pointed out repeatedly, the marginal cost of distributing that electronic information to one more person is as close to zero as humanity has ever seen. No, you don’t have to because copyright is always on your side, but if you don’t, you open yourself up to being undercut by the following business model:
…which is exactly what ControlsOverload is. Remember in my last post where I said that I took 3 minutes of my own time and posted the question and answer related to my problem at ControlsOverload? Check out what happens when you search for compactlogix type 01 code 01 fault or even compactlogix powerup fault. The first item in the search results is my question and answer. You can even go to that page and add more information, or correct inconsistencies and you don’t even need to sign-up!
Rockwell has one group that makes products and another group that charges people for support on those products. In order to protect the revenue model of the second, they put barriers in place to maintain a monopoly on information, even though every single industry that has relied on maintaining a monopoly on access to information is dying a slow miserable death. Meanwhile I wasted another hour of my time trying to get a product I’d already purchased to work, and they had the information to cut that time down to 2 minutes, but they put all these barriers in my way even though I’d already paid for support!
The price to hire a PLC programmer is anywhere from about $50 to $100 an hour in my experience. Let’s say $75. Every time Rockwell’s support barriers waste an hour of my time, the total cost of ownership of their equipment goes up by, on average, $75. On top of that, it frustrates me enough to blog about it, and there’s an opportunity cost as well. I could have been using that hour to improve the efficiency of the machine in some other way rather than fighting with Rockwell’s internet site.
Here’s my suggestion to Rockwell to fix the situation:
What do I mean by paid premium content? Tutorials. Training videos. Example applications. Code review services (a support person can review your automation program and offer advice). E-books. Insider news & tips. Access to beta versions of upcoming software releases. In short, something I’m willing to pay for above and beyond what I believe I’ve already paid for.
Good luck Rockwell – the next few years are going to be interesting.
I’ve decided to create a tutorial for beginners getting started with RSLogix 5000 from Rockwell Automation. Part 1 is already posted: Creating a New Project. I will be filling in the rest over the next few weeks. I hope new automation graduates and experienced members of the automation industry who are migrating from another platform will both find something useful.
As always, I welcome any and all constructive feedback.
I recently had a problem with an Allen-Bradley CompactLogix processor. The power went out, came back on, and the processor had faulted with a major fault, type 01, code 01. The fault message said “Power lost and restored in RUN mode”. There must be a way to disable this fault, and just have it go back into run mode so the operator can recover.
I Googled for the fault message and I got a link to a helpful forum thread. In that forum thread there was a link to a Rockwell Automation Knowledgebase Article that seemed to have the information I was looking for. I clicked the link and it told me I needed to login. That’s annoying enough, but fine. I tried my normal username and password for such sites, and that didn’t work. I went into my encrypted file and pulled out the username and password I’d saved for Rockwell Automation. That didn’t work.
Ok, fine. It had an option to email me my user name. I waited a few minutes for the email – yes the username I was using was OK. I clicked on the option to reset my password. Got the email, reset the password to the one I had saved anyway. Success! I was logged in, but it had forgotten what article I was trying to get to, so I went back to the original forum post and clicked the link again. “This article has been locked, or …” blah blah blah about a TechConnect ID. Ok, so I go to my profile page and click on the tab for TechConnect support IDs. None registered. Hmm, no obvious way to register one. I go to the other profile page… TechConnect ID… excellent! I enter that, click save, and now it needs my company name, address, etc. Ok, I enter that… but now I can’t click save. I have to go backwards, re-enter the TechConnect ID, and my company name and address, and then it saved. Ok, great!
I go back to the forum and click the link again. “This article has been locked, or …” What?!? I check my list of TechConnect IDs again, and there’s nothing there. This is really annoying me at this point. Oddly, under my name it now has the name of some Mexican company, and some place in Mexico. I check my profile page again and it all looks right, and I double-check the TechConnect ID. It’s right (and I know it’s right because I’ve used it to call Rockwell tech support recently).
I go back to Google and search again for the fault code, and I see a link to the Rockwell knowledge-base article. I click on that… same message! What’s going on here? How can Google even index a page that’s behind a sign-up wall and doesn’t even show you the page unless you have a valid TechConnect ID?
Going back to the original forum post, I did find some useful information in there, but obviously the knowledge-base article would have helped me the most. How far behind is Rockwell Automation’s online support? They’re still in the 90’s. I realize there are still a lot of people in this industry who are happier to pickup a phone and call their phone support in a situation like this, but as time goes on and the Millennial Generation continues entering the workforce, self-help focused people like myself are going to be more and more commonplace. We won’t settle for getting our questions answered in hours or even 30 minutes anymore; we want to solve our stupid problems like this in minutes or even seconds, and the technology is there to let us do it. Stop putting unnecessary barriers in our way! Rockwell Automation online support: FAIL!
By the way, I just spent 3 extra minutes and entered my technical question and answer over on ControlsOverload, a website for technical questions and answers about industrial automation. A website where you don’t have to sign in, and nothing is ever blocked behind any kind of wall. If Google can see it, so can you. This is the future of finding information on the internet. Rockwell Automation: Make it easy for me to use your hardware, and maybe I’ll buy more! You know what… here’s Beckhoff’s whole 350MB knowledge-base available for download so you can take it with you onsite when you don’t have an internet connection. Brilliant isn’t it? It’s called openness and it’s the new name of the game. Wake up!
Update (9 MAR 2010): The Global Quality Leader from Rockwell has contacted me and it looks like we’re going to have a constructive discussion about some of these issues (and he also gave me a different TechConnect ID to try). +1 to Rockwell for having their ears on!
Update (20 MAR 2010): I posted a write-up on Rockwell’s response here.
I’ve been working on an HMI application recently (RSView ME). At the end of the day I saved everything, closed the project, then used Application Manager to backup the application to an .apa file and store it on the server.
This morning I opened the local copy of the application, and a significant portion of the work I had done yesteday was “undone”. It was only one of the displays, but all of the changes were missing. I restored a copy of the backup, and it was the same there. Now I’m pretty paranoid about saving my work, so I probably saved that display about 50 times yesterday, and it was definitely saving somewhere. The question was, where?
Here’s what I did to get myself into this problem:
- Started by copying one display (let’s say “A”) to make a new one (let’s say “B”).
- I then renamed the duplicate (let’s say from “B” to “C”). I probably didn’t close the display before renaming it (which might have caused the problem).
- I made all of my changes to the display, saving along the way.
It turns out that the changes I made were being saved against “B”, not “C” even though “B” didn’t really exist anymore. I was able to recover from this problem with the following procedure:
- Close the application (but leave RSView Studio open if you want)
- Goto C:\Documents and Settings\All Users\Documents\RSView Enterprise\ME\HMI projects\[proj-name]\Gfx
- Rename C.Gfx to D.Gfx
- Rename B.Gfx to C.Gfx
- Open the application again in RSView Studio. The changes should be restored.
I hope that helps someone one day. 🙂
Recently someone asked me what the difference was between PLC programming and PC programming. This was a topic I’ve talked about before, and I gave him the same old answer I’d already come up with, but it got me thinking about it again.
Now I have a new way to compare PC programming and PLC programming: PC programming is like brick-laying and PLC programming is like sculpting.
When you write a PC program, you start out by writing the least amount of code you can to get a functional program. You run and debug that. You check it in to source control, and then you repeat the cycle, building a little more, testing it, and at regular intervals you show it to a customer, get feedback, and change direction, but you always work in small increments because it’s easier to build it that way. You make sure you’ve built a solid foundation, and then you build more on top.
On the contrary, the first time you download a PLC program to a new machine, you have a fixed number of outputs that you have to account for. You can’t just program one axis of motion first without taking into account the position of other axes on the machine (unless you enjoy the sound of metal deforming). So you start by writing logic for all the outputs and you load on more and more conditions that prevent the motion from occurring until you’re absolutely sure that nothing will move unless it’s absolutely safe. Of course, when you download this program to the machine, nothing moves, no matter what buttons you push. This is actually a good place to be. In some cases you actually put conditions in that you know could never be true, just to make sure nothing will move until you’re really sure you want it to.
In a letter from 1549, Michelangelo defined sculpture as the art of “taking away” not that of “adding on” (the process of modeling in clay), which he deemed akin to painting. (Reference)
Now that you’ve downloaded your logic to the PLC, debugging and commissioning the machine is largely a matter of taking away those conditions that you didn’t really need. This is done one motion at a time until the machine does exactly what you want, when you want, but only that. Your PLC program is done, not when there’s nothing left to add, but when there’s nothing left to take away.
That is, until you add a data collection system…