Author Archives: Scott Whitlock

About Scott Whitlock

I'm Scott Whitlock, an "automation enthusiast". By day I'm a PLC and .NET programmer at ETBO Tool & Die Inc., a manufacturer.

Beckhoff in the Kitchen

Leave a reply

I was a little surprised when I read about Industrial PC Technology in High End Kitchens in Beckhoff’s PC Control magazine. The entire kitchen is integrated together using industrial PC technology:

SieMatic’s self-developed software for the selection menu and the control functions forms the heart of the SieMatic Grid. The basis is a
modularly structured Beckhoff Embedded PC equipped with a Windows XP Embedded operating system as well as a .Net framework. Via RS232, the Beckhoff PC controls the T+A components as well as managing all functions that they do not take care of. Above all, these naturally include all the typical PC tasks, such as Internet access, e-mail access and data services (such as weather forecast or newsfeeds), but also entertainment programs, such as slide shows. All Miele@home appliances are also integrated via the Ethernet interface of the Embedded system.

The most important functions, however, run invisibly in the background, because the Beckhoff PC works almost like a mediator between different worlds: for example, a television broadcast can be interrupted for messages from the Miele system – that the dishwasher program has finished, for instance – if the user so desires. The integration of other systems – for example, the picture from an IP monitoring camera – is easy to accomplish using Beckhoff hardware. It was clear from the outset to SieMatic project manager Thorsten Pawelczyk that the SieMatic Grid would be a constantly growing and changing system: “The fact that the Beckhoff automation system hardly sets any limits regarding the integration of other systems comes just at the right time.”

I question the logic of putting industrial technology in a consumer kitchen, but I must admit I’m impressed and envious. I think the real question is, does it have the “hackability” we’ve come to expect from industrial control systems? Can I connect to it with TwinCAT PLC and modify the functions, extend it, or tie in new systems? Somehow I doubt it. The article mentions it runs a .NET application, so in this case they’re probably only using the hardware, not the Beckhoff control software.

How Many Times Must I Tell You, “Don’t Repeat Yourself”?

Leave a reply

I spend a significant amount of my time these days doing PC programming (as opposed to PLC programming) and I’d say the most time consuming part of my job is following the DRY Principle, aka “Don’t Repeat Yourself”.

The naive interpretation of the DRY principle is that it’s about minimizing the amount of code you write. While this may sometimes be a side effect, this isn’t the point of DRY at all. What we’re trying to do is structure the program so that there’s just one place for everything, and everything is in it’s one place. This is a lot harder than it sounds because no matter how you segment your program into modules, classes, or layers there’s always something that cuts across those boundaries.

Take logging for instance. Every part of the application needs to log things to the same log file. The naive implementation of opening a file, writing a line, and closing it is short but doesn’t follow the DRY principle because we’re repeating this simple sequence of steps all over the place. The danger is that if we change our logging strategy, we now have to change multiple places in every file.

There are probably two ways that the “logging strategy” could change:

  1. We change where (or if) we want to log the data, like to a database instead of a file
  2. We change the kinds of stuff we want to log

We can handle the first case easily by moving the logging logic into a global subroutine, or if you’re more advanced you would use something like a logging service with the service locator pattern. That would let us change where we log the data, and if we include a severity level with the logging interface, we could filter our logging by severity.

The second case is really hard to solve. We have to choose each spot in the code where we want to log information. Maybe every time I catch an error in a try/catch block, but that means every time I write a try/catch block I have to add a call to my logging service. This case violates the DRY principle because the idea is just “every time I catch an error in a try/catch block”. That idea needs to be codified in one place. Ideally I should be able to extend the try/catch block of the language and say, “any time you execute a catch I also want you to do this”. That’s not possible with .NET, that I know of anyway.

Likewise, maybe I want to know every time a user carries out any action that might affect the underlying data in the database, and I’ll have to add logging code at each of those places too. If you have all of your database access going through a single layer in your application, at least you can confine the logging to that layer, but it may consist of dozens of classes representing dozens of database tables. That’s where we need something like the ADO.NET entity framework, and have all of our database entities derive from a single base class, and then we might be able to tie our logging rules in there.

The problem is that these architectural solutions are hard. They take a lot of effort, and when you just need to add logging, it’s too easy to just go through your code and add logging wherever you need it, and you’ve created a maintenance nightmare. It’s the extremely low cost of copy and paste vs. architecture that creates this problem. If you had to pay $1 every time you copied and pasted code, I bet you’d write more maintainable software.

The Invisible Hand: Replication

Leave a reply

What makes the masses adopt a new technology? I don’t think it’s because it saves them time.

When I was at StackOverflow Dev Days last year, Joel Spolsky gave a keynote that touched on the topics of why people use technology. Specifically he talked about why the masses use it. I think as geeks we sometimes can’t see how “normal people” see technology. Back in the 90’s I couldn’t understand why anyone would buy a Mac over a PC. As a 30-something father with different priorities, I’m starting to “get it”.

Joel referred to a book by Richard Dawkins called The Selfish Gene. It says that genes aren’t the method by which we reproduce, but rather: we are the method by which genes reproduce. They created us because it’s a workable method to make more copies of themselves.

Joel asks us to imagine a 20 year old college student sitting in her dorm room, trying to install Firefox. Why is she installing Firefox? So that she can use Facebook. Why does she want to use Facebook? So she can communicate with her friends and know where they’re going to be, possibly when they’ll be going to the bar. And why does she want to go to the bar? Most likely there’s a strong primal urge to “hook up” with someone.

People buy iPhones for the sexiness. Business people want a BlackBerry for business, but why are they in business? Why do business people want to be successful?

It doesn’t really matter how you try to spin this. Once you get outside the early adopters of technology who are doing it because they’re “not normal”, the “normal” people use technology for much more fundamental reasons. The most powerful force that drives everything, including technology, is replication, and thus reproduction. Do you want proof?

Did you notice how replication was its own reward? Makes sense. Things that just replicate themselves will make more things that replicate themselves. Things that don’t replicate themselves will get eaten by those that do.

Now ask yourself, why hasn’t home automation taken off? Lights? Vacuums? They’re gadgets for a geeky minority. They really don’t help with the fundamental drive of the home, which is family. How does home automation help you raise better kids? How does it help you make them smarter, better, more successful? Does this home automation system appeal to the vast majority of “normal” people? I don’t think so. (I’ll give it some geek cred, though.)

If we answered those questions, we’d know how to make home automation for the masses.

On Time

Leave a reply

I have to say that this newspaper article from 1992 predicted our future right on the money. I’m an “automation enthusiast”, so I’m all about the time saving device. Just for fun I entered “time saving device” into Google and clicked “I’m Feeling Lucky”. I was directed to this question:

If every year time saving devices are invented (i.e. computers, cell phones, email, etc…) then why do we still need a 40 hour work week?

As I’ve said before, this is another manifestation of the Jevons Paradox. If I wanted to hire someone to paint a building and I had to choose between someone with a bucket of paint and a paintbrush, or a guy with a power sprayer, the latter is undoubtedly more expensive per hour, but cheaper per square foot of wall. He’ll also have the job done faster. That means the demand for the guy with the sprayer is higher. His time saving device gives him more work and less time.

Saving Time is the Myth of Home Automation

In Industrial Automation, we never really talk about saving time. We talk about saving labor and saving money. We talk about increased productivity, efficiency, throughput and uptime. Time is a constant.

Yet at home we talk about time like it’s a variable. As if you could have more of it. What an odd concept when you think about it…

I’ve recently spent some time trying to think of a good time saving home automation device to create, and I’m convinced it’s a pointless activity. Would an automatic pet feeder save me time? Possibly, but you still have to refill it, the parts are likely to wear out, and it has to be cleaned regularly. It takes one of us less than 30 seconds to feed the dog every day. That’s about 3 hours per year. But if you spent 10 minutes a month cleaning it, that’s 2 hours a year right there, and how long are you going to spend fiddling with it when it finally breaks down? Even if there was no extra time, would you spend $100 to save 3 hours? It’s marginal, but doubtful.

Now where that pet feeder really shines is if you have a dog with a special need. It can feed your animal up to 8 times a day, so if your dog was diabetic, then you can start to see that it solves a difficult problem really effectively.

Would a robot vacuum save me time? I think you’ll find that it fails in the same way that the pet feeder fails: you have to clean out the vacuum bucket every time it cycles, and you have to take it apart to clean the brushes regularly. The batteries need replacing, and most reviews I’ve seen indicate that you still need to vacuum once every couple of weeks. It doesn’t save you time. It gives you the ability to vacuum more often.

Look at the three successful categories of home automation: HVAC, lighting, and audio/video. None of those are about saving you time. They’re all about improving the (perceived) quality of life. A new robot that folded your laundry for you wouldn’t be as successful as a closet that color co-ordinates your outfits for you. Everyone can fold laundry but not everyone can pick an outfit. There’s a market for the latter.

On Helping

Leave a reply

If I want something from you, I’ll probably tell you what you want to hear.

If I want to help you, I’ll probably tell you something you don’t want to hear.

Isn’t it interesting that we prefer to be around people who want something from us, rather than people who want to help us?

Phidgets!

Leave a reply

I got ahold of some Phidgets on the weekend. These are basically USB I/O devices for amateur robotics, but I’m looking at their use in the home automation space.

Phidgets Interface Kit 8/8/8
They have drivers for lots of operating systems, and APIs for almost any programming platform under the sun.

Getting the I/O connected and controlled from a .NET application was a breeze, including the Interface Kit 8/8/8 and the little R/C Servo Controller. That little servo could certainly move a damper in a heating duct, and they have lots of environmental sensors. The wheels have started turning…

On Readability

6 Replies

Programming both PCs and PLCs sometimes gets me thinking about programming from a higher level. I’ve written a lengthy answer over on StackOverflow about the differences between PC and PLC programming. What I haven’t talked about before is how they are the same.

First, let me define what I mean by PC and PLC programming. By PC programming, I’m generally referring to imperative programming. There are actually two popular PC programming paradigms: imperative and declarative, and the paradigm with new-found popularity, functional, is actually a subset of declarative programming.

How PC and PLC programming are NOT the same

Most PLC programming falls into the declarative category, and most PC programming falls into the imperative category. For examples:

PC:

  • Visual Basic, C/C++, C#: imperative
  • Lisp, F#: functional
  • HTML, XAML: declarative

PLC:

  • Ladder logic: declarative
  • Function block diagram: functional
  • Structured text: imperative

So normally when we talk about the differences between PC and PLC programming, we’re talking about the differences between imperative and declarative programming, but there’s obviously overlap on both sides of the fence.

The major difference, however, is audience. In North America at least, we write PC programs with the expectation that other programmers will have to read, understand and make changes to them, but we write PLC programs with the expectation that people in the maintenance department will be expected to go online with and troubleshoot our programs. Just think about how odd that would be in the PC world: when a word processor crashes, nobody whips out their debugger, figures out what caused the program to crash, makes a fix and continues writing their letter. Primarily this is because the source code doesn’t come with the word processor, but it’s also because the programming language can only be understood by programmers.

How PC and PLC programming ARE the same

When you look at what makes a PC program good or bad, on a high level it’s the same thing that makes a PLC program good or bad: readability. Now as I’ve pointed out, the people who have to read the program is different in each case, but really readability is the fundamental measure by which experienced programmers rate programs.

On the PC side ,the name of the game with readability is modularity. You want to divide up your program into parts, and you want to make those individual parts as self-contained as possible. You want to minimize the interaction to these parts as much as possible. That makes it easier to reason about the program because you’re abstracting away the underlying complexity on each piece and leaving a less complex interface that you can interact with. The entire domain of object oriented programming is an extension of the concept of modularity.

On the PLC side, readability is equivalent to being able to troubleshoot the machine when it’s down. Experienced PLC programmers ask themselves, “if this machine stopped unexpectedly and I had to figure out why it stopped, what would I do? How can I make it easier for someone following that process to figure out what’s wrong with the machine?”

It turns out that most people troubleshooting a machine follow a similar procedure: you start at the outputs and you work your way backwards. You generally have a good idea what the machine is supposed to do next (e.g. move slide A to position B). You can look at the print set, or even the valve itself and figure out what output should be turning on. You look at the indicator on the output card and it’s not on, so the logic isn’t telling it to turn on. You crack open the laptop, and you find that output. You’re looking for one thing: the COIL.

Notice the one big mistake you could make if you’re writing a program: you could use a whole bunch of set and reset (or latch and unlatch) instructions to drive your outputs. Based on my description, you can easily see why that would make the program less readable: which set instruction is the one that’s supposed to be turning on the output right now? If there’s only one, it’s easy, but if there are 10, you’re already lost.

Let’s assume you do find the coil that drives this output. Your next step is to follow the logic back through the rungs, right clicking on the conditions that aren’t satisfied and cross referencing until you understand what the machine is waiting for. What are some obvious mistakes you can make that would hinder this process?

  • Using an integer (or sequencer – yuck!) to store your automatic process step number rather than using individual coils for each step
  • Using set/reset or latch/unlatch instructions more than once on each bit
  • Using really long tag names so readers have to scroll left/right or up/down more than necessary to read one rung
  • Calling subroutines more than once per scan so you can’t see the state of the logic in the subroutine (newer controllers have function blocks where you can drill down into individual instances, which is nice)
  • Using For Loops – same reason
  • Having logic that is conditionally scanned – particularly in controllers where it isn’t obvious if the logic you’re looking at is scanned or not
  • Mapping your inputs or outputs by block copying them to or from a user defined type, word or array (Don’t make the reader start counting bits! The line is down!)

Once you start thinking from the point of view of someone troubleshooting the machine, your perspective on good vs. bad programming really changes. You realize that techniques that seem to save you time while you’re programming end up costing the company hours of lost production time while maintenance picks their way through your cryptic logic.

Next time you’re writing your ladder logic, think of the poor maintenance guy who has to figure out what’s wrong, and try to make his life a little less miserable.

Automation Vendors on Twitter

Leave a reply

I’ve noticed some Industrial Automation vendors on Twitter recently (not to mention some automation related magazines).

I like that trend, but here’s some commentary on who’s doing what right and wrong (from an engineer’s perspective anyway):

Tweets with useful information for customers:

Tweets aimed at shareholders, not customers:

Not taking it seriously:

An aborted attempt:

Rockwell Automation Responds

6 Replies

After my recent rant about Rockwell Automation and their frustrating online support, I received an email from Joe Harkulich, Global Quality Leader at Rockwell Automation. The first thing he did was, helpfully, get me straightened out with my TechConnect ID. Apparently the one we were given wasn’t our real TechConnect ID, and it happened to be the ID of another company in Mexico, which caused some confusion on my end. Once that got straightened out, I could access the article I was looking for (a bit late, but better than never).

Joe then invited me to join a conference call with Esther Beris (Rockwell Knowledgebase), Jon Furniss (TechConnect), and Rob Snyder (Senior Manager for Rockwell’s remote support). I’m thankful for them taking the time to do this, as I’m really only a moderate user of Rockwell Automation products. (I really don’t think they’re very concerned about my legions of blog readers and twitter followers). I’ll relate a few of the details of the call here, and I really want to express my gratitude to everyone involved for their time.

The first item on the agenda was Joe addressing some misinformation in my previous blog post. I had made a comment about Beckhoff’s 350MB knowledge-base available for download, and I had suggested that Rockwell should do something similar. It turns out that Rockwell does offer TechConnect subscribers the ability to receive 3 DVD’s full of all their product manuals and their entire knowledge base. Joe offered to send me a copy, and I accepted. I received them yesterday and installed them on the laptop at work. This will be nice pending an on-site trip that’s scheduled next month. I’m a bit concerned that the discs are dated May of 2008. I was expecting them to be about 6 months old. We did discuss the idea of making them automatically download updates of the knowledge base, etc., so your offline copy is always up to date, but they pointed out that this would be a huge amount of information so it probably couldn’t happen, but they would take the idea into consideration. Personally I think a solution like that would have a lot of value.

We then turned our discussion towards some of the issues I had raised with Rockwell’s online support in my previous post. I certainly made it clear that I have always been impressed with Rockwell’s paid telephone support, and I really have nothing to complain about there; they are simply awesome! When it comes to the online support, we talked about these issues:

  • Single sign-on
  • Expiry of your account if you don’t log on for 6 months
  • Access to product manuals, EDS files and support directly from the product page
  • Paid vs. unpaid content

Single Sign-On

It makes no sense to me why you have to sign on separately at www.rockwellautomation.com and at their knowledgebase. Apparently it doesn’t make a lot of sense to the folks at Rockwell either, and they’re working on it. However, there’s no time-line for the resolution of it (or at least they couldn’t give me one). I think it’s one of those cases where the politics of a big organization are getting in the way of doing something really simple, and I can appreciate that. I did get the impression that the people involved do care, and it will get done, eventually.

Expiry of your account if you don’t log on for 6 months

In past years I would frequently do a lot of automation work, and then switch to PC programming for a project or two, and then switch back to automation work as I’ve recently done. I always found it a bit frustrating the first time I tried to download some EDS file and Rockwell had canceled my account because it had been longer than 6 months since I’d signed in. Apparently I’m not the only customer to complain about this, and Rockwell is in the process of changing this. Bravo!

Access to product manuals, EDS files and support directly from the product page

Let’s say I go to the ControlLogix Product Page. On a normal website when I look at a product I see exactly what that one component looks like, a link directly to the datasheet or user manual, a tab with all the specs on it, and all the information I need to buy that product. But the ControlLogix page is a brand page, not a product page. It’s almost completely useless to me as an Engineer. There is a Literature link, and that has a selection guide and something else, but then it tells me I have to go to the literature library to find all the good documentation.

When I explained this on the conference call I got some good laughs, because again, I’m not the first person to complain about this. Apparently this is being resolved, and they do have a time-line: September 2010. That’s great news and I’m really happy to hear it.

Paid vs. Unpaid Content

I’m afraid the issue of paid vs. unpaid content is still a sticking point for me. They explained to me that they chose a paid model for their support because they can provide better support. There’s no question in my mind that the quality of Rockwell’s telephone support comes through time and again, but I have to disagree with the pay-wall model they’re using for online support. I really think what’s happened is that the technical support group within Rockwell is literally scared of what could happen if they didn’t maintain a strangle-hold over the information that guys like me need to do their job.

I’ve talked before about vendor lock-in in the industrial automation industry, so I’m not going to go into that again. But this is a case where a company is selling me upwards of $10,000 of automation equipment per project and once they’ve done that, they want to charge me extra for access to the information I need to make it work the way it was supposed to work in the first place. I understand the paid telephone support model because there’s a one-to-one relationship between the amount of time I spend on the phone and the amount of time they have to pay someone to talk on the phone. But while they’re doing that, they’re already creating a knowledge-base of information. As smarter people than I have pointed out repeatedly, the marginal cost of distributing that electronic information to one more person is as close to zero as humanity has ever seen. No, you don’t have to because copyright is always on your side, but if you don’t, you open yourself up to being undercut by the following business model:

  • Start an online knowledge base where the customers can build it
  • Offer subtle rewards to the people who contribute (peer recognition within the knowledge community)
  • Let Google index the site and drive traffic to it

…which is exactly what ControlsOverload is. Remember in my last post where I said that I took 3 minutes of my own time and posted the question and answer related to my problem at ControlsOverload? Check out what happens when you search for compactlogix type 01 code 01 fault or even compactlogix powerup fault. The first item in the search results is my question and answer. You can even go to that page and add more information, or correct inconsistencies and you don’t even need to sign-up!

Rockwell has one group that makes products and another group that charges people for support on those products. In order to protect the revenue model of the second, they put barriers in place to maintain a monopoly on information, even though every single industry that has relied on maintaining a monopoly on access to information is dying a slow miserable death. Meanwhile I wasted another hour of my time trying to get a product I’d already purchased to work, and they had the information to cut that time down to 2 minutes, but they put all these barriers in my way even though I’d already paid for support!

The price to hire a PLC programmer is anywhere from about $50 to $100 an hour in my experience. Let’s say $75. Every time Rockwell’s support barriers waste an hour of my time, the total cost of ownership of their equipment goes up by, on average, $75. On top of that, it frustrates me enough to blog about it, and there’s an opportunity cost as well. I could have been using that hour to improve the efficiency of the machine in some other way rather than fighting with Rockwell’s internet site.

Here’s my suggestion to Rockwell to fix the situation:

  • Create and harness a money-free and barrier-free user generated content site, just like ControlsOverload, where customers can help other customers
  • Keep the excellent telephone support, but have the support people contribute to and maintain the free site rather than the paid site
  • Move all the paid support information to the free site (this doesn’t include premium content, as described below)
  • Use the free site to advertise premium online content that users like me are willing to pay for

What do I mean by paid premium content? Tutorials. Training videos. Example applications. Code review services (a support person can review your automation program and offer advice). E-books. Insider news & tips. Access to beta versions of upcoming software releases. In short, something I’m willing to pay for above and beyond what I believe I’ve already paid for.

Good luck Rockwell – the next few years are going to be interesting.