Tag Archives: plc

Beckhoff TwinCAT: a First Impression

25 Replies

Those of you who know me remember I’ve been itching to get my hands on a Beckhoff TwinCAT PLC industrial control system for a few years now, but I never found a project or customer willing to take on the risk of trying something new. Like a bunch of penguins on a cliff of ice, nobody wanted to be the first in the water.

I’ve recently had the opportunity to apply a TwinCAT PLC and HMI system to a relatively low-risk application, so I finally have something to share. Those of you who know me also know I don’t pull my punches when I talk about products, so here’s an honest review of the TwinCAT system: the good, the bad, and the ugly.

Benefits

Cost

What draws you to TwinCAT is its combination of low cost with a ton of features. I can’t publish prices, but I encourage you to compare prices from Beckhoff, Allen-Bradley, Omron, and others. Beckhoff offers a similar or better feature-set, with as many or more communication options than it’s competitors, for significantly less money (it’s real-time PC control instead of dedicated hardware). On specs it can do everything, and doesn’t bend you over the table when it comes to the price.

Scalability

First, Beckhoff TwinCAT PLC takes advantage of a PC CPU and memory, which means you can scale up the performance of your system for a fraction of the cost of scaling up to large PLC and PAC systems.

Secondly, if you use EtherCAT, you get to scale up your I/O sizes without as much impact on performance or price. The individual I/O slices are significantly cheaper than for other bus technologies, and the servo drives are less expensive because the bus speed is fast enough to close the loop from the PLC rather than in the device itself.

Where TwinCAT really shines is when you need to integrate it to the MES layer in your organization. Most other vendors require you to buy expensive OPC or other connection software, but Beckhoff provides a free DLL for Windows applications to communicate with the PLC over a communication protocol called ADS. (They also provide an OPC server as an add-on if you need it.)

What’s the Catch

Switching software vendors is never easy. If you’re used to your existing PLC software like RSLogix, you’ve got a bit of a learning curve ahead of you. Here are the things that down-right annoyed me about TwinCAT when I was coming from the RSLogix/PanelView realm:

  • When you do an online edit, it resets all of your forces. You have to reload your watch list to re-apply them. (Fortunately I always write perfect code the first time, so I never had to do an online change…)
  • The ladder logic editor doesn’t seem to auto-wrap (unless I’m missing a setting somewhere)
  • The HMI editor crashed a couple of times when I was modifying the HMI, and I had to restart the TwinCAT system (not the PC, just the software)
  • The English documention is lacking
  • The TwinCAT System Manager DeviceNET master configuration had a bug where it wouldn’t recognize EDS files where the Class value was stored as a 16 bit, rather than an 8 bit value. The EDS file was valid, according to EZ-EDS, but it wouldn’t read it. Changing the value to 8-bit made it recognize the parameter. This made it hard, but not impossible, to integrate an Allen-Bradley Point I/O DeviceNET node.
  • Even though the HMI will log an alarm history, there’s no alarm history control for you to view it in the HMI itself (it’s saved to a CSV file)

Here are some things that are just really different, but I’ve had no trouble adapting to:

  • You have to define all variables (tags) in a text syntax, instead of in a spreadsheet-like tag editor
  • I/O isn’t mapped directly. You define your hardware I/O and your program I/O and then you map them.
  • You don’t do online changes while you’re online with the PLC. You “log off”, then make your change, and then “log on” and tell it to upload your change. (You can do this while the machine is running.)

What’s Cool

Using a PC based controller opened up some interesting features, particularly in the HMI:

  • You can launch another application from a button click on the HMI (Excel, etc.)
  • You can configure some interesting alarm actions, like launching a program, or sending an email
  • There’s an (optional) add-in to let the PLC communicate in a limited fashion with a SQL Server
  • It runs on commodity hardware. PC hardware is cheap and you can get much larger touch screens for a lot less than the cost of an equivalent PanelView. The “fieldbus” card (EtherCAT) is under $100. Even a Beckhoff EtherCAT to DeviceNET master solution is cheaper than a straight-up DeviceNET card from another manufacturer.
  • TwinSAFE – a Safety Controller sitting right on your fieldbus for less than a stand-alone safety controller from another vendor.

Support

I had nothing but superb support from the local Beckhoff technical support guys during this experience. They respond to emails personally usually within a couple of hours, and they have no problem taking the two hour drive to see us if we have any issues that can’t be resolved over the phone or with email. We’re not a “big” customer in any way.

Summary

I’ve just spent about 2 weeks coming up to speed on TwinCAT, having come from an AB/Omron background, with a smattering of other PC based controllers like Phoenix Contact’s Visual Logic Controller (VLC) and MultiProg (formerly PCWorx).

The ease-of-use of TwinCAT doesn’t measure up to RSLogix, but it’s comparable to Omron, and it blows products like PCWorx/MultiProg out of the water (I think I was using version 3 of PCWorx, and I know there’s a version 5 now). Most of my issues with TwinCAT are that its ladder editor just isn’t polished enough compared with RSLogix, but that’s not surprising given its roots in Germany. North America tends to favor ladder logic, where Europe seems to favor instruction list, structured text, and function block diagram. It feels like the TwinCAT ladder logic editor was neglected a bit, but it is certainly workable. Comparing the TwinCAT sequential function chart editor with the RSLogix sequential function chart editor, they were actually reasonably close. I liked that I didn’t have to position individual steps in TwinCAT – it just auto-arranged everything for me.

Price-wise, TwinCAT is significantly less expensive than any other serious player in the industrial automation equipment space, particularly when you start ramping up the amount of I/O and drives you need, and it has similar (and in some cases better) features to offer. It’s even significantly less expensive than the Phoenix Contact solution (from my memory), and given that, there’s no reason I’d ever recommend a PCWorx/MultiProg solution from Phoenix Contact. The Phoenix offering is worse and more expensive.

If you’re in a position where your facility is in the market for a new automation platform, I definitely recommend including the Beckhoff TwinCAT platform in your deliberations. It hits a really unique sweet-spot on price, flexibility, and features that nobody else seems to be able to offer right now. Now that we’re a few weeks in, we have a running machine, it looks like the project will be successful and I have no reservations about saying: if you don’t mind learning something new, give them a try.

Automation and Software in the Next Decade

2 Replies

The “low intellectual property” Fashion Industry as a model for innovation?

I have to agree. Since I’ve started getting involved in open source software, I can tell you that intellectual property laws are the biggest obstacle to innovation that I face. I can say that because I took the time to consult with intellectual property lawyers on everything from trademark to copyright to open source license to software patent. Here’s what I found out: you can’t write a single line of code without violating someone’s patent, somewhere. However, the only time that person will enforce their patent claim is if you’re really successful. The only defense is to pay tens or hundreds of thousands of dollars to build up a patent portfolio of your own so you can partake in a ludicrous arms-race deterrent. On top of that, your patent portfolio offers no defense against so-called “patent trolls”: companies that only own patents and don’t write software of their own. They’re the equivalent of a terrorist cell with nuclear weapons… your nukes are ineffective because you don’t have a city to target.

Copyright is under control, mostly because it’s so much weaker than patents, and it’s free. It stops whole-sale copying of your software, but allows someone to look at your source code, learn from it, and then write it themselves without violating copyright (if they’re careful). Plus, the wide adoption of open source licenses give us lots of material that we can freely copy or integrate.

The irony is that patents were supposed to speed up innovation, but they’ve done exactly the opposite in the software industry.

Do you want to know why none of this matters though? I’m going to make a bold prognostication here:

If the last decade (have we agreed on what to call it yet? The Oh’s?) was about more crap, then the next decade will be about customized crap. Seriously, we gorged ourselves on the mountain of crappage that is sold at Wal-mart and people are starting to wake up with a hangover. We’ve already seen the desires start to change from more to better but I think we’re going to see it change to mine. People are going to want unique stuff. Things that they had a hand in customizing. Something they can use to express themselves.

This has major implications for manufacturing, so it affects the automation industry, and I think it also affects software. Just like the fashion industry, where people use clothes to express themselves, we’re about to enter an age when all the crap you sell to consumers has to have a story, a personality, and be unique. Mass production will remain for your basic staples, and China is going to continue to provide us with that, but North America and Europe are going to have to start producing bespoke crap.

What does this mean for automation? We’re going to see a rise in the build-a-bear style build-your-own-product-on-a-website and have it manufactured and sent to your door the next day. Imagine the logistical changes that need to take place to make that happen. Internet merchandising companies like cafepress already have a leg up in this department. Their merchandise is manufactured after you order it. Now imagine a product with 10,000 or a million different variations, and the automation that has to support it. Imagine the automation required to manufacture products whose specifications change as fast as fashion trends.

On the software side, we’re going to see every major application support custom 3rd party add-ins so everyone can customize their software so it works for them. I think we’re also going to see a rise in “pseudo-programming languages” that let people who aren’t programmers actually customize their applications in ways that we’ve never let them do before. Platforms that give people the power to build solutions to their own needs are going to flourish, and one-size-fits-all solutions like word processors will fade into obscurity.

So I think we’ll have two options: the vast majority of us are going to spend our time building custom things for individuals or businesses. The successful ones will produce products that people can customize themselves.

A Bigger Picture of Automation

2 Replies

When you’re immersed in the world of industrial automation for a long time, you may start to think that PLCs, relays and valves are automation. That’s really not the case. Automation applies to doing anything automatically. Those little timers you plug your lights into when you go on vacation are automation. Automatically billing your customer as soon as product is scanned when it goes on the truck is automation.

To me, we’ve solved a lot of the excruciatingly difficult technical challenges, like motion control, because they can be boiled down to academic pursuits and packaged into a black box. The pain is in the integration. The tyranny of dry contact I/O and 4-20 mA analog signals is because they’re universal standards that nobody’s figured out how to patent yet (or the patents have lapsed). These universal standards make integration of components from various manufacturers possible, and it’s the backbone that supports the industrial automation industry.

The leap from the mechanical and electrical to the digital world should have made integration easier and more flexible. To some extent it has, but we now have these proprietary walls built between our solutions. I once wouldn’t hesitate to connect a valve from company A to a relay from company B, and those companies knew it, and the competition kept prices down and drove innovation forward every year. Now I have to make design decisions about which vendor’s hardware plays better with others before I can buy the first item on my bill of materials. What kind of valve bank do I buy? It depends on whether or not I need a ProfiBus or a DeviceNET communication module on it.

There’s no doubt that the digital alternatives are cheaper than hard wired solutions, and we have made progress on standard bus systems thanks to organizations like the ODVA, but what we really lack is a common open file format for storing our automation programs. As long as we’re stuck with the legacy traditional PLCs, then automation itself is stuck in the mud. But there are open automation systems right around the corner. Systems that aren’t going to be focused on industrial automation, and they’re going to tackle the integration problem in a way that proprietary systems just can’t do for us.

I’m building one of these next-generation systems right now. It’s an automation platform. It’s open. It’s free. It’s not a PLC.

On Readability

6 Replies

Programming both PCs and PLCs sometimes gets me thinking about programming from a higher level. I’ve written a lengthy answer over on StackOverflow about the differences between PC and PLC programming. What I haven’t talked about before is how they are the same.

First, let me define what I mean by PC and PLC programming. By PC programming, I’m generally referring to imperative programming. There are actually two popular PC programming paradigms: imperative and declarative, and the paradigm with new-found popularity, functional, is actually a subset of declarative programming.

How PC and PLC programming are NOT the same

Most PLC programming falls into the declarative category, and most PC programming falls into the imperative category. For examples:

PC:

  • Visual Basic, C/C++, C#: imperative
  • Lisp, F#: functional
  • HTML, XAML: declarative

PLC:

  • Ladder logic: declarative
  • Function block diagram: functional
  • Structured text: imperative

So normally when we talk about the differences between PC and PLC programming, we’re talking about the differences between imperative and declarative programming, but there’s obviously overlap on both sides of the fence.

The major difference, however, is audience. In North America at least, we write PC programs with the expectation that other programmers will have to read, understand and make changes to them, but we write PLC programs with the expectation that people in the maintenance department will be expected to go online with and troubleshoot our programs. Just think about how odd that would be in the PC world: when a word processor crashes, nobody whips out their debugger, figures out what caused the program to crash, makes a fix and continues writing their letter. Primarily this is because the source code doesn’t come with the word processor, but it’s also because the programming language can only be understood by programmers.

How PC and PLC programming ARE the same

When you look at what makes a PC program good or bad, on a high level it’s the same thing that makes a PLC program good or bad: readability. Now as I’ve pointed out, the people who have to read the program is different in each case, but really readability is the fundamental measure by which experienced programmers rate programs.

On the PC side ,the name of the game with readability is modularity. You want to divide up your program into parts, and you want to make those individual parts as self-contained as possible. You want to minimize the interaction to these parts as much as possible. That makes it easier to reason about the program because you’re abstracting away the underlying complexity on each piece and leaving a less complex interface that you can interact with. The entire domain of object oriented programming is an extension of the concept of modularity.

On the PLC side, readability is equivalent to being able to troubleshoot the machine when it’s down. Experienced PLC programmers ask themselves, “if this machine stopped unexpectedly and I had to figure out why it stopped, what would I do? How can I make it easier for someone following that process to figure out what’s wrong with the machine?”

It turns out that most people troubleshooting a machine follow a similar procedure: you start at the outputs and you work your way backwards. You generally have a good idea what the machine is supposed to do next (e.g. move slide A to position B). You can look at the print set, or even the valve itself and figure out what output should be turning on. You look at the indicator on the output card and it’s not on, so the logic isn’t telling it to turn on. You crack open the laptop, and you find that output. You’re looking for one thing: the COIL.

Notice the one big mistake you could make if you’re writing a program: you could use a whole bunch of set and reset (or latch and unlatch) instructions to drive your outputs. Based on my description, you can easily see why that would make the program less readable: which set instruction is the one that’s supposed to be turning on the output right now? If there’s only one, it’s easy, but if there are 10, you’re already lost.

Let’s assume you do find the coil that drives this output. Your next step is to follow the logic back through the rungs, right clicking on the conditions that aren’t satisfied and cross referencing until you understand what the machine is waiting for. What are some obvious mistakes you can make that would hinder this process?

  • Using an integer (or sequencer – yuck!) to store your automatic process step number rather than using individual coils for each step
  • Using set/reset or latch/unlatch instructions more than once on each bit
  • Using really long tag names so readers have to scroll left/right or up/down more than necessary to read one rung
  • Calling subroutines more than once per scan so you can’t see the state of the logic in the subroutine (newer controllers have function blocks where you can drill down into individual instances, which is nice)
  • Using For Loops – same reason
  • Having logic that is conditionally scanned – particularly in controllers where it isn’t obvious if the logic you’re looking at is scanned or not
  • Mapping your inputs or outputs by block copying them to or from a user defined type, word or array (Don’t make the reader start counting bits! The line is down!)

Once you start thinking from the point of view of someone troubleshooting the machine, your perspective on good vs. bad programming really changes. You realize that techniques that seem to save you time while you’re programming end up costing the company hours of lost production time while maintenance picks their way through your cryptic logic.

Next time you’re writing your ladder logic, think of the poor maintenance guy who has to figure out what’s wrong, and try to make his life a little less miserable.

Rockwell Automation Responds

6 Replies

After my recent rant about Rockwell Automation and their frustrating online support, I received an email from Joe Harkulich, Global Quality Leader at Rockwell Automation. The first thing he did was, helpfully, get me straightened out with my TechConnect ID. Apparently the one we were given wasn’t our real TechConnect ID, and it happened to be the ID of another company in Mexico, which caused some confusion on my end. Once that got straightened out, I could access the article I was looking for (a bit late, but better than never).

Joe then invited me to join a conference call with Esther Beris (Rockwell Knowledgebase), Jon Furniss (TechConnect), and Rob Snyder (Senior Manager for Rockwell’s remote support). I’m thankful for them taking the time to do this, as I’m really only a moderate user of Rockwell Automation products. (I really don’t think they’re very concerned about my legions of blog readers and twitter followers). I’ll relate a few of the details of the call here, and I really want to express my gratitude to everyone involved for their time.

The first item on the agenda was Joe addressing some misinformation in my previous blog post. I had made a comment about Beckhoff’s 350MB knowledge-base available for download, and I had suggested that Rockwell should do something similar. It turns out that Rockwell does offer TechConnect subscribers the ability to receive 3 DVD’s full of all their product manuals and their entire knowledge base. Joe offered to send me a copy, and I accepted. I received them yesterday and installed them on the laptop at work. This will be nice pending an on-site trip that’s scheduled next month. I’m a bit concerned that the discs are dated May of 2008. I was expecting them to be about 6 months old. We did discuss the idea of making them automatically download updates of the knowledge base, etc., so your offline copy is always up to date, but they pointed out that this would be a huge amount of information so it probably couldn’t happen, but they would take the idea into consideration. Personally I think a solution like that would have a lot of value.

We then turned our discussion towards some of the issues I had raised with Rockwell’s online support in my previous post. I certainly made it clear that I have always been impressed with Rockwell’s paid telephone support, and I really have nothing to complain about there; they are simply awesome! When it comes to the online support, we talked about these issues:

  • Single sign-on
  • Expiry of your account if you don’t log on for 6 months
  • Access to product manuals, EDS files and support directly from the product page
  • Paid vs. unpaid content

Single Sign-On

It makes no sense to me why you have to sign on separately at www.rockwellautomation.com and at their knowledgebase. Apparently it doesn’t make a lot of sense to the folks at Rockwell either, and they’re working on it. However, there’s no time-line for the resolution of it (or at least they couldn’t give me one). I think it’s one of those cases where the politics of a big organization are getting in the way of doing something really simple, and I can appreciate that. I did get the impression that the people involved do care, and it will get done, eventually.

Expiry of your account if you don’t log on for 6 months

In past years I would frequently do a lot of automation work, and then switch to PC programming for a project or two, and then switch back to automation work as I’ve recently done. I always found it a bit frustrating the first time I tried to download some EDS file and Rockwell had canceled my account because it had been longer than 6 months since I’d signed in. Apparently I’m not the only customer to complain about this, and Rockwell is in the process of changing this. Bravo!

Access to product manuals, EDS files and support directly from the product page

Let’s say I go to the ControlLogix Product Page. On a normal website when I look at a product I see exactly what that one component looks like, a link directly to the datasheet or user manual, a tab with all the specs on it, and all the information I need to buy that product. But the ControlLogix page is a brand page, not a product page. It’s almost completely useless to me as an Engineer. There is a Literature link, and that has a selection guide and something else, but then it tells me I have to go to the literature library to find all the good documentation.

When I explained this on the conference call I got some good laughs, because again, I’m not the first person to complain about this. Apparently this is being resolved, and they do have a time-line: September 2010. That’s great news and I’m really happy to hear it.

Paid vs. Unpaid Content

I’m afraid the issue of paid vs. unpaid content is still a sticking point for me. They explained to me that they chose a paid model for their support because they can provide better support. There’s no question in my mind that the quality of Rockwell’s telephone support comes through time and again, but I have to disagree with the pay-wall model they’re using for online support. I really think what’s happened is that the technical support group within Rockwell is literally scared of what could happen if they didn’t maintain a strangle-hold over the information that guys like me need to do their job.

I’ve talked before about vendor lock-in in the industrial automation industry, so I’m not going to go into that again. But this is a case where a company is selling me upwards of $10,000 of automation equipment per project and once they’ve done that, they want to charge me extra for access to the information I need to make it work the way it was supposed to work in the first place. I understand the paid telephone support model because there’s a one-to-one relationship between the amount of time I spend on the phone and the amount of time they have to pay someone to talk on the phone. But while they’re doing that, they’re already creating a knowledge-base of information. As smarter people than I have pointed out repeatedly, the marginal cost of distributing that electronic information to one more person is as close to zero as humanity has ever seen. No, you don’t have to because copyright is always on your side, but if you don’t, you open yourself up to being undercut by the following business model:

  • Start an online knowledge base where the customers can build it
  • Offer subtle rewards to the people who contribute (peer recognition within the knowledge community)
  • Let Google index the site and drive traffic to it

…which is exactly what ControlsOverload is. Remember in my last post where I said that I took 3 minutes of my own time and posted the question and answer related to my problem at ControlsOverload? Check out what happens when you search for compactlogix type 01 code 01 fault or even compactlogix powerup fault. The first item in the search results is my question and answer. You can even go to that page and add more information, or correct inconsistencies and you don’t even need to sign-up!

Rockwell has one group that makes products and another group that charges people for support on those products. In order to protect the revenue model of the second, they put barriers in place to maintain a monopoly on information, even though every single industry that has relied on maintaining a monopoly on access to information is dying a slow miserable death. Meanwhile I wasted another hour of my time trying to get a product I’d already purchased to work, and they had the information to cut that time down to 2 minutes, but they put all these barriers in my way even though I’d already paid for support!

The price to hire a PLC programmer is anywhere from about $50 to $100 an hour in my experience. Let’s say $75. Every time Rockwell’s support barriers waste an hour of my time, the total cost of ownership of their equipment goes up by, on average, $75. On top of that, it frustrates me enough to blog about it, and there’s an opportunity cost as well. I could have been using that hour to improve the efficiency of the machine in some other way rather than fighting with Rockwell’s internet site.

Here’s my suggestion to Rockwell to fix the situation:

  • Create and harness a money-free and barrier-free user generated content site, just like ControlsOverload, where customers can help other customers
  • Keep the excellent telephone support, but have the support people contribute to and maintain the free site rather than the paid site
  • Move all the paid support information to the free site (this doesn’t include premium content, as described below)
  • Use the free site to advertise premium online content that users like me are willing to pay for

What do I mean by paid premium content? Tutorials. Training videos. Example applications. Code review services (a support person can review your automation program and offer advice). E-books. Insider news & tips. Access to beta versions of upcoming software releases. In short, something I’m willing to pay for above and beyond what I believe I’ve already paid for.

Good luck Rockwell – the next few years are going to be interesting.

The RSLogix 5000 Tutorial

Leave a reply

I’ve decided to create a tutorial for beginners getting started with RSLogix 5000 from Rockwell Automation. Part 1 is already posted: Creating a New Project. I will be filling in the rest over the next few weeks. I hope new automation graduates and experienced members of the automation industry who are migrating from another platform will both find something useful.

As always, I welcome any and all constructive feedback.

Does Rockwell Automation Hate its Customers?

I recently had a problem with an Allen-Bradley CompactLogix processor. The power went out, came back on, and the processor had faulted with a major fault, type 01, code 01. The fault message said “Power lost and restored in RUN mode”. There must be a way to disable this fault, and just have it go back into run mode so the operator can recover.

I Googled for the fault message and I got a link to a helpful forum thread. In that forum thread there was a link to a Rockwell Automation Knowledgebase Article that seemed to have the information I was looking for. I clicked the link and it told me I needed to login. That’s annoying enough, but fine. I tried my normal username and password for such sites, and that didn’t work. I went into my encrypted file and pulled out the username and password I’d saved for Rockwell Automation. That didn’t work.

Ok, fine. It had an option to email me my user name. I waited a few minutes for the email – yes the username I was using was OK. I clicked on the option to reset my password. Got the email, reset the password to the one I had saved anyway. Success! I was logged in, but it had forgotten what article I was trying to get to, so I went back to the original forum post and clicked the link again. “This article has been locked, or …” blah blah blah about a TechConnect ID. Ok, so I go to my profile page and click on the tab for TechConnect support IDs. None registered. Hmm, no obvious way to register one. I go to the other profile page… TechConnect ID… excellent! I enter that, click save, and now it needs my company name, address, etc. Ok, I enter that… but now I can’t click save. I have to go backwards, re-enter the TechConnect ID, and my company name and address, and then it saved. Ok, great!

I go back to the forum and click the link again. “This article has been locked, or …” What?!? I check my list of TechConnect IDs again, and there’s nothing there. This is really annoying me at this point. Oddly, under my name it now has the name of some Mexican company, and some place in Mexico. I check my profile page again and it all looks right, and I double-check the TechConnect ID. It’s right (and I know it’s right because I’ve used it to call Rockwell tech support recently).

I go back to Google and search again for the fault code, and I see a link to the Rockwell knowledge-base article. I click on that… same message! What’s going on here? How can Google even index a page that’s behind a sign-up wall and doesn’t even show you the page unless you have a valid TechConnect ID?

Going back to the original forum post, I did find some useful information in there, but obviously the knowledge-base article would have helped me the most. How far behind is Rockwell Automation’s online support? They’re still in the 90’s. I realize there are still a lot of people in this industry who are happier to pickup a phone and call their phone support in a situation like this, but as time goes on and the Millennial Generation continues entering the workforce, self-help focused people like myself are going to be more and more commonplace. We won’t settle for getting our questions answered in hours or even 30 minutes anymore; we want to solve our stupid problems like this in minutes or even seconds, and the technology is there to let us do it. Stop putting unnecessary barriers in our way! Rockwell Automation online support: FAIL!

By the way, I just spent 3 extra minutes and entered my technical question and answer over on ControlsOverload, a website for technical questions and answers about industrial automation. A website where you don’t have to sign in, and nothing is ever blocked behind any kind of wall. If Google can see it, so can you. This is the future of finding information on the internet. Rockwell Automation: Make it easy for me to use your hardware, and maybe I’ll buy more! You know what… here’s Beckhoff’s whole 350MB knowledge-base available for download so you can take it with you onsite when you don’t have an internet connection. Brilliant isn’t it? It’s called openness and it’s the new name of the game. Wake up!

Update (9 MAR 2010): The Global Quality Leader from Rockwell has contacted me and it looks like we’re going to have a constructive discussion about some of these issues (and he also gave me a different TechConnect ID to try). +1 to Rockwell for having their ears on!

Update (20 MAR 2010): I posted a write-up on Rockwell’s response here.

PLC Programming is like Sculpting

2 Replies

Recently someone asked me what the difference was between PLC programming and PC programming. This was a topic I’ve talked about before, and I gave him the same old answer I’d already come up with, but it got me thinking about it again.

Now I have a new way to compare PC programming and PLC programming: PC programming is like brick-laying and PLC programming is like sculpting.

When you write a PC program, you start out by writing the least amount of code you can to get a functional program. You run and debug that. You check it in to source control, and then you repeat the cycle, building a little more, testing it, and at regular intervals you show it to a customer, get feedback, and change direction, but you always work in small increments because it’s easier to build it that way. You make sure you’ve built a solid foundation, and then you build more on top.

Cross-leg Slave by MichaelangeloOn the contrary, the first time you download a PLC program to a new machine, you have a fixed number of outputs that you have to account for. You can’t just program one axis of motion first without taking into account the position of other axes on the machine (unless you enjoy the sound of metal deforming). So you start by writing logic for all the outputs and you load on more and more conditions that prevent the motion from occurring until you’re absolutely sure that nothing will move unless it’s absolutely safe. Of course, when you download this program to the machine, nothing moves, no matter what buttons you push. This is actually a good place to be. In some cases you actually put conditions in that you know could never be true, just to make sure nothing will move until you’re really sure you want it to.

In a letter from 1549, Michelangelo defined sculpture as the art of “taking away” not that of “adding on” (the process of modeling in clay), which he deemed akin to painting. (Reference)

Now that you’ve downloaded your logic to the PLC, debugging and commissioning the machine is largely a matter of taking away those conditions that you didn’t really need. This is done one motion at a time until the machine does exactly what you want, when you want, but only that. Your PLC program is done, not when there’s nothing left to add, but when there’s nothing left to take away.

That is, until you add a data collection system…

Lowering the Cost of Automation Equipment

5 Replies

(This is the third part of a trilogy of blog posts about what I think is the biggest roadblock preventing significant growth in the industrial automation industry. You should read Part 1: Industrial Automation Technology is Stagnant and Part 2: Why Automation Equipment Vendors Dabble in Integration first.)

I was recently making significant additions to a PLC program and the resulting program was too big to fit in the existing PLC. There weren’t many areas where the code could be made smaller without impacting readability, so I went out and priced the same series of PLC with more memory. I’m not going to name brands or distributors here (and I’d probably get in a lot of trouble if I did, because they don’t publish their prices), but I was taken aback by one thing: the only difference between these two controllers was the amount of memory (execution speed and features were the same), and it was over $1000 more. I realize this is “industrial” equipment, so the memory probably has increased temperature ratings, and it’s battery backed, but how much PLC memory can you buy for $1000? As of writing this post, the cost of 1 Gibabyte of memory for a desktop computer is less than $50.

At $50 per Gig, 20 Gigs of PLC memory? Nope.

Ok, it’s industrial, so… 1 Gig? Nope.

Try less than half a Megabyte. Yes.

Just for the sake of comparison, how much is that per Gigabyte? If half a Meg costs $1000, then the price of PLC memory is 2 Million dollars per Gigabyte. WTF? It’s not 40 times more expensive, it’s forty thousand times more expensive than commodity RAM.

Do you know why they charge two million dollars per gigabyte for PLC memory? Vendor lock-in. It’s that simple. They can charge thousands of dollars because the switching costs are insanely high. I can either buy a PLC in the same series from the same manufacturer (and they make sure there’s only one distributor in my area) with more memory, or I can replace the PLC with one from another manufacturer incurring the following costs:

  • Rewriting the program because the languages are not portable.
  • Modifying the electrical drawings significantly.
  • Rewiring the electrical panel.
  • Retraining all of my existing maintenance and engineering staff on the new platform.

So it really doesn’t matter. They could charge us $5000 for that upgrade and it’s still cheaper than the alternative. Why don’t they publish their prices? Simple, they price it by how much money they think you have, not by how much it costs them to produce the equipment. Wouldn’t we all like to work in an industry like that?

How did it get this way? In the PC industry, if Dell starts charging me huge markups on their equipment, I can just switch to another supplier of PC equipment. Even in the industrial PC world, many vendors offer practically equivalent industrial PCs. You can buy a 15″ panel mount touch screen PC with one to two gigs of RAM for about $3000. Add $500 and you can probably get an 80 Gig solid state hard drive in it. Compare that with the fact that small manufacturing plants are routinely paying $4500 for a 10″ touch screen HMI that barely has the processing power to run Windows CE, and caps your application file at a few dozen megabytes!

The difference between the PC platform and the PLC platform comes down to one thing: interoperability standards. The entire explosive growth of the PC industry was based on IBM creating an open standard with the first IBM PC. Here’s the landscape of personal computers before the IBM PC (pre 1980)*:

Here’s the market share shortly after introduction of the IBM PC (1980 to 1984):

… and here’s what eventually happened (post 1984):

Nearly every subsystem of a modern PC is based on a well defined standard. All motherboards and disk drives use the same power connectors, all expansion cards were based on ISA, or later PCI, and AGP standards. External devices used standard RS232, and later USB ports. Hardware that wasn’t interoperable just didn’t survive.

That first graph is exactly like where we are now in the automation industry with PLCs. None of the vendors have opened their architecture for cloning, so there’s no opportunity for the market to commoditize the products. Let’s look at this from the point of view of a PLC manufacturer for a moment. Opening up your platform to cloning seems like a really risky move. Wouldn’t all of your customers move to cheaper clones? Wouldn’t it drive down the price of your equipment? Yes, those things would happen.

But look at what else would happen:

  • The hardware cost of your platform would drop, so everyone would leave your competitor’s platforms and come to yours. At least you’d still be in the game, but your competitors wouldn’t even have a compatible platform anymore.
  • By dropping the price of automation equipment, the demand for complementary products would increase. IBM didn’t make its fortune directly in the PC industry, but they rode the wave and became an enterprise services company built on top of the commodity PC business.
  • The size of the industry would explode.

Why don’t we have open standards? What about IEC-61131-3? PLC vendors tell us that their controllers are IEC 61131-3 compliant, but that doesn’t mean they’re compatible. The standard only specifies what languages must be supported and what basic features those languages should have. Why didn’t the committee at least specify a common file format? You may be surprised to learn that many of the major committee members were actually from the PLC equipment manufacturers themselves. The same people who have a vested interest in maintaining their own vendor lock-in were the ones designing the standard. No wonder!

What about the new XML standard? Well, it’s being specified by the same group and if you read their documentation, you can clearly see that this XML standard is not meant for porting programs between PLC brands, but rather to integrate the PLC programming tools with 3rd party programs like debugging, visualization, and simulation tools. You can be certain that a common vendor-neutral file format will not be the result of this venture.

Solutions

I see three ways this problem could be solved:

  1. An association of common manufacturers and system integrators could form and develop a truly open and interoperable standard,
  2. A PLC manufacturer could “go IBM” and open up their platform, or
  3. A single integrator or manufacturer could develop and create an open automation platform and put the rights in the public domain.

Option 1 is almost doomed to fail because it has the word committee in it, but options 2 and 3 (which are similar) have happened before in other industries and stand a good chance of success here.

In my opinion, the most likely candidate to fulfill option 2 is Beckhoff. I say this because they’ve already opened up their EtherCAT fieldbus technology as an open standard, and their entire strategy is based around leveraging existing commodity hardware. Their PLC is actually a real-time runtime on a PC based system, and their EtherCAT I/O uses commodity Ethernet chipsets. All they would really need to do is open source their TwinCAT I/O and TwinCAT PLC software. Their loss of software license revenue could be balanced by increased demand for their advanced software libraries in the short term, and increased demand for their EtherCAT industrial I/O in the longer term. Since none of the other automation vendors have a good EtherCAT offering, this could launch Beckhoff into the worldwide lead relatively quickly.

For option 3, any company that considers automation equipment to be an expense or a complementary product (i.e. manufacturing plants and system integrators) could do this. There is a long term ROI here by driving down the cost of automation equipment. Many internet companies do this on a daily basis. IBM sells servers and enterprise services, which is the natural complement of operating systems, so they invest heavily in Linux development (not to drive down the cost of Linux – it’s free – but to offer a competitive alternative to Microsoft to keep Microsoft’s software costs down). Google does everything it can to increase internet usage by the public, so it invests heavily in free services like Google Maps, Gmail, and even the Firefox web browser. The more people who surf, the more people who see Google’s advertising, and the more money they make.

If I were going to go about it, I’d build it like this:

  • Base it on commodity PC hardware (x86).
  • Pick an open source, free real time operating system like FreeRTOS.
  • Document and publish (free download) the communication protocol and file format for the automation program files (e.g. ladder logic).
  • Write a reference runtime that conforms to this communication protocol and publish it under a BSD license. It doesn’t have to be great, but it has to work and be useful.
  • Write an extensible programming environment (Windows-based) where you can develop automation programs that conform to the standard and that communicates with any runtime that conforms to the standard. Give it away free and publish it under the GNU GPL license to prevent it from being embraced and extended in a proprietary way by one vendor.

If you did that, anyone could make hardware that’s compatible with this platform, people are free to innovate on the runtime, and anyone can make improvements to the programming environment (as long as they give their changes back to the community). Instant de-facto standard.

I know some of you may be grinning right now because this is the path I’m following with my next personal software project, that I’m calling “Snap” (which stands for “Snap is Not A PLC”). However, Snap is for the home automation industry, not the industrial automation industry. If any company out there wants to take up the gauntlet and do this for industrial automation equipment, you may just change this industry and solve our biggest problem.

Discussion is welcome. Please comment below or contact me by email (e8pzrhs02@sneakemail.com).

* Graphs courtesy of ars technica.

Dogs, Cats and Moody Machines

Leave a reply

Have you ever wondered what the ladder logic for a dog would look like?  I imagine it’s something like this:

ladder-logic-for-dog

… and so on.  I think that’s why we consider dogs so loyal.  Perhaps by loyal, we mean predictable, or understandable.

Ever wonder what the ladder logic for a cat would look like?  I imagine it thus:

ladder-logic-for-cat

… or something like that.  Actually I’m pretty sure I’ve met a couple of cats that came equipped with thirteen sequencers and a conditional subroutine jump in there somewhere.  It certainly makes life interesting.  Does that little tail wag mean it’s safe to pet, or does it mean your cat is about to mistake your inner thigh for prey?  Who knows!  What fun!

I guess my point is, cats can be moody, and believe it or not, so can machines.  You might call it “internal machine state”, but I call it moodiness.  Have you ever been trying to troubleshoot a machine and it was stuck thinking there was a part in one of the stations that really wasn’t there?  Every time it indexes it keeps faulting?  That’s machine moodiness.  So there you are, flagging every sensor in sight trying to get that part present bit to clear, and no matter how many roses or chocolates you buy for the darned thing, you know you’ll be sleeping in the dog house tonight.

Thankfully, there’s a cure for machine moodiness:  Make all internal state visible and editable.  At the very least, there should be a screen on the HMI that shows the current status of the part present bits at each station.  If you really want to be fancy, make sure it allows the operator to set and clear those bits manually.  That includes latches, sealed coils, counters, FIFOs, and even long running timers.

Anyway, if you can’t see it, you can’t troubleshoot it, so adding visibility will save you time in the future.  Trust me.  And trust dogs; they’re quite loyal.