Category Archives: Automation

Good Function Blocks, Bad Function Blocks

3 Replies

In case you’ve never read my blog before, let me bring you up to speed:

  • Write readable PLC logic.

Now, I’m a fan of ladder logic, because when you write it well, it’s readable by someone who isn’t a programmer, and (in North America, anyway) maintenance people frequently have to troubleshoot automation programs and most of them are not programmers.

That doesn’t mean I’m not a fan of other automation languages. I think structured text should be used when you’re parsing strings, and I like to use sequential function chart to describe my auto-mode logic. I’m also a fan of function block diagram (FBD), particularly when working with signal processing logic, like PID loops, etc.

What I’m not a fan of is hard-to-understand logic. Here’s FBD used wisely:

Here’s an example of FBD abuse:

I’m still reading Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin. He’s talking about traditional PC programming, but one of the “rules” he likes to use is that functions shouldn’t have many inputs. Ideally 0 inputs, maybe 1 or 2, possibly 3, but never more than 3. He says if you go over 3, you’re just being lazy. You should just break that up into multiple functions.

I think that applies equally well to FBD. The reader can easily rationalize about the first image, above, but the second one is just a black box with far too many inputs and outputs. If it doesn’t work the way you expect (and it’s doubtful it does), you have to keep going inside of it to figure it out. Unfortunately once you’re inside, all the variable names change, etc.

I understand the necessity of code re-use, but not code abuse. If you find yourself in the situation of example #2, ask yourself how you can refactor it into something more readable. After all, the most likely person who has to read this later is you.

SoapBox Snap is Finally Released

3 Replies

At long last, I’ve finally released a first version of SoapBox Snap, a free and open source ladder logic editor and runtime for your PC:

SoapBox Snap

After all this time, it’s finally ready? Yes. I’m admittedly a perfectionist, and yet I admit this software isn’t perfect. Software never is, really. But in the world of open source software, the question is, “is it useful?” I believe SoapBox Snap is already useful, even in this early form. You can write logic that controls your outputs, which is the entire point of this software. More features will come over time.

One really cool feature, that I’m glad made it into the first release, is a driver for Twitter. You can easily configure SoapBox Snap to watch your own Twitter feed for status updates. The driver exposes a “Last Status” signal that has the text of your last status and a signal that pulses any time your status changes. You can compare the status text to certain key phrases, like “turn on light”, and use it to drive outputs. I think it would be neat to tie it to your garage door opener, and if you forget your keys you can just pull out your phone and tweet some key phrase like, “darn it I forgot my keys!” and the garage door could open!

Please check it out, download it, and play with it. If you have any neat ideas of things you could do, please drop me a line. Have fun!

Clean Ladder Logic

1 Reply

I’ve recently been reading Clean Code: A Handbook of Agile Software Craftsmanship. It’s written by Robert C. “Uncle Bob” Martin of Agile software (among other) fame. The profession of computer programming sometimes struggles to be taken seriously as a profession, but programmers like Martin are true professionals. They’re dedicated to improving their craft and sharing their knowledge with others.

The book is all about traditional PC programming, but I always wonder how these same concepts could apply to my other obsession, ladder logic. I’m the first to admit that you don’t write ladder logic the same way you write PC programs. Still, the concepts always stem from a desire for Readability.

Martin takes many hard-lined opinions about programming, but I think he’d be the first to admit that his opinions are made to fit the tools of the time, and those same hard-and-fast rules are meant to be bent as technology marches on. For instance, while he admits that maintaining a change log at the top of every source file might have made sense “in the 60’s”, the rise of powerful source control systems makes this obsolete. The source control system will remember every change that was made, who made it, and when. Similarly, he advocates short functions, long descriptive names, and suggests frequently changing the names of things to fit since modern development environments make it so easy to rename and refactor your code.

My favorite gem is when Martin boldly states that code comments, while sometimes necessary, are actually a failure to express ourselves adequately in code. Sometimes this is a lack of expressiveness in the language, but more often laziness (or pressure to cut corners) is the culprit.

What would ladder logic look like if it was “clean”? I’ve been visiting this question during the development of SoapBox Snap. For instance, I think manually managing memory, tags, or symbols is a relic of older under-powered PLC technology. When you drop a coil on the page in SoapBox Snap, you don’t have to define a tag. The coil is the signal. Not only is it easier to write, it prevents one of the most common cardinal sins of beginner ladder logic programming: using a bit address in two coil instructions.

Likewise, SoapBox Snap places few if any restrictions on what you can name your coils. You don’t have to call it MTR1_Start – just call it Motor 1: Start. Neither do you need to explicitly manage the scope of your signals. SoapBox Snap knows where they are. If you drop a contact on a page and reference a coil on the same page, it just shows the name of the coil, but if you reference a contact on another page, it shows the “full name” of the other coil, including the folders and page names of your organization structure to find it. Non-local signals are obviously not local, but you still don’t have to go through any extraneous mapping procedure to hook them up.

While we’re on the topic of mapping, if you’ve read my RSLogix 5000 Tutorial then you know I spend a lot of time talking about mapping your inputs and your outputs. This is because RSLogix 5000 I/O isn’t synchronous. I think it’s pointless to make the programmer worry about such pointless details, so SoapBox Snap uses a synchronous I/O scan, just like the old days. It scans the inputs, it solves the logic, and then it scans the outputs. Your inputs won’t change in the middle of the logic scan. To me, fewer surprises is clean.

I’ve gone a long way to make sure there are fewer surprises for someone reading a ladder logic program in SoapBox Snap. In some ladder logic systems, the runtime only executes one logic file, and that logic file has to “call” the other files. If you wanted to write a readable program, you generally wanted all of your logic files to execute in the same order that they were listed in the program. Unfortunately on a platform like RSLogix 5000, the editor sorts them alphabetically, and to add insult to injury, it won’t let you start a routine name with a number, so you usually end up with routine names like A01_Main, A02_HMI, etc. If someone forgets to call a routine or changes the order that they execute in the main routine, unexpected problems can surface. SoapBox Snap doesn’t have a “jump to page” or “jump to routine” instruction. It executes all logic in the order it appears in your application and each routine is executed exactly once per scan. You can name the logic pages anything you want, including using spaces, and you can re-order them with a simple drag & drop.

Program organization plays a big role in readability, so SoapBox Snap lets you organize your logic pages into a hierarchy of folders, and it doesn’t limit the depth of this folder structure. Folders can contain folders, and so on. Folder names are also lenient. You can use spaces or special characters.

SoapBox Snap is really a place to try out some of these ideas. It’s open source. I really hope some of these innovative features find their way into industrial automation platforms too. Just think how much faster you could find your way around a new program if you knew there were no duplicated coil addresses, all the logic was always being executed, and it’s always being executed in the order shown in the tree on the left. The productivity improvements are tangible.

Off-the-Shelf or Custom Automation?

Leave a reply

If you’re like me, you’re a fan of customizing:

…and certainly in the automation industry you see a lot of custom control solutions. In fact there’s always been this long-running debate over the value of custom solutions vs. the value of off-the-shelf “black box” products.

I’ve noticed this rule: the closer you get to the production line, the more custom things you’ll see. Just look at the two ends of this extreme: production lines are almost always run by PLCs with custom logic written specifically for that one line, but the accounting system is almost always an off-the shelf product.

There’s a good reason for this. Accounting methodologies are supposed to be standardized across all companies. Businesses don’t claim that their value proposition is their unique accounting system (unless you’re talking about Enron, I suppose). Automation, however, is frequently part of your business process, and business processes are the fundamental business proposition of a company. Fedex should definitely have a custom logistical system, Amazon needs to have custom order fulfillment, and Google actually manufactures their own servers. These systems are part of their core business strengths.

So when should a company be buying off-the-shelf automation solutions? I say it’s any time that “good enough” is all you need. You have to sit down and decide how you’re going to differentiate yourself from the competition in the mind of your customers, and then you have to focus as much energy as possible on achieving that differentiation. Everything else needs to be “good enough”. Everything else is a cost centre.

If you follow that through logically, it means you should also seek to “commoditize” everything in the “everything else” category. That bears repeating: if it’s not a core differentiator for your company, you will benefit if it becomes a commodity. That means if you have any intellectual property sitting there in a non-critical asset, you should look for ways to disseminate that to the greater community. This is particularly important if it helps the industry catch up to a leading competitor.

There are lots of market differentiators that can depend on your automation: price, distribution, and quality all come to mind. On the other hand there are other market differentiators that don’t really depend on your automation, like customer service or user-friendly product designs. Ask yourself what category your company fits in, and then you’ll know whether custom automation makes sense for you.

Quick thoughts about Automation

Leave a reply

I think that once you’ve been in this industry for a few years, you need to reach out to others to share some of the wisdom you’ve learned. Most of the knowledge we carry around can do other people a lot more good than it will do us again in the future, so sharing needs to be a cultural norm. With that thought in mind, here are some quick automation-related thoughts I’d like to share:

  1. Inexperienced engineers appear to work faster, but their solutions are less maintainable. [tweet this]
  2. Choose open systems over proprietary, when possible [tweet this]
  3. Automate your own job ruthlessly before to automate anything else. It pays back. [tweet this]
  4. Beware of employers who spend 30 minutes reprimanding you about a 15 minute line on your timesheet. [tweet this]
  5. If you can’t find a more powerful tool, make your own. [tweet this]
  6. When estimating a project, if you’re counting in hours, you’re not being realistic. Use half-days. [tweet this]
  7. Don’t take shortcuts writing a program if it’s at the expense of readability. It doesn’t pay off. [tweet this]
  8. Automation doesn’t help if you don’t understand the process you’re automating. [tweet this]
  9. Blame is reactive. “What can we do differently next time?” is proactive. [tweet this]
  10. Innovate is a verb. This is not a coincidence – it requires constant action. [tweet this]
  11. Make things of value, not emails. [tweet this]

Feel free to share your own nuggets of wisdom below.

The “Almost There” Paradox

3 Replies

We’re all probably familiar with the idea that it takes half the time to get to 90% done and the other half to finish the last 10%. This is a staple of project management.

I think there’s actually a narrower scope of really dangerous solutions that you only become familiar with after you experience it. There’s a whole set of problems where the obvious solution gets you 95 to 98% of the way to your performance spec really quickly, but is almost impossible to reach 100% by incremental improvements. The reason I say they’re dangerous is because the feeling of being “almost there” prevents you from going back to the drawing board and coming up with a completely parallel solution.

I can remember a machine vision job from years ago where the spec was “100% read rate”. I only got it to about 94%, and someone else gave it a try. He got it up over 96%, but 100% was out of reach given the technology we had.

Experiences like that make you conservative. Now I unconsciously filter possible solutions by their apparent “flakiness”. I’m much more likely to add an extra prox to a solution to verify a position than to rely on timers or other kinds of internal state, because the latter are more prone to failure during system starts and stops. I press for mechanical changes when I used to bend under the pressure to “fix it in software”.

Still, you have to be careful. Its easy to discount alternatives just because they bear some passing resemblance to a bad experience you had before. You have to keep re-checking your assumptions. Unfortunately, rapid prototyping usually fails to uncover the “almost there” situation I’m talking about. If you prototype something up fast, and it works in 97% of your lab tests, you’ll probably think you have a “proof of concept”, and go forward with it.

The best way to test new solutions is to put them into production on a low risk system. If you’re an integrator, this means having a really good relationship with your customer (chances are you need their equipment to run your tests). If you work for a manufacturer, you can usually find some out-of-the-way machine to test on before you go all-in.

“Best Practices,” Indeed

3 Replies

I’ve just been reading Ken McLaughlin’s recent post Top Ten Signs an Integrator is the Real Deal #7: Best Practices and Standards and I have to say, my initial reaction is one of skepticism. I think Ken’s thinking is a little too narrow on this one. Let me explain…

This isn’t the first time I’ve considered the “problem of standards” on this blog. In an earlier post, Standards for the Sake of Standards, I explained how most corporate standards eventually end up being out-of-date and absurd, mostly because nobody making the standard ever things to write down Why the standard exists, which would allow future policy-makers to understand the reasons and change the standard when it no longer applied. Instead, it becomes gospel.

However, that isn’t to say you could run a large organization without best practices and standards. That’s the point isn’t it? In order to become large, you need built-in efficiency, and you do that at the expense of innovation. Big companies don’t innovate (in fact the only notable exception is Apple, and the rebuttal is always, “fine, so give one example other than Apple”). Almost all innovation happens in small companies, by a tightly knit group of superstars where the chains have been removed. Best Practices are, in fact, put in place to clamp down on innovation because innovation is risky, and investors hate risk. It’s better to make lots of average product for average people than exceptional products for a few people (hence McDonald’s). Paul Graham, as usual, has something insightful to add to this:

Within large organizations, the phrase used to describe this approach is “industry best practice.” Its purpose is to shield the pointy-haired boss from responsibility: if he chooses something that is “industry best practice,” and the company loses, he can’t be blamed. He didn’t choose, the industry did.

I believe this term was originally used to describe accounting methods and so on. What it means, roughly, is don’t do anything weird. And in accounting that’s probably a good idea. The terms “cutting-edge” and “accounting” do not sound good together. But when you import this criterion into decisions about technology, you start to get the wrong answers.

The reason small companies are innovative is that innovative people can’t stand corporate environments. Imagine if you were an inspired chef… could you stand working at McDonald’s? Could McDonald’s even stand to employ you? You’d be too much trouble! You’d have to work in that nice one-off restaurant called “Maison d’here” where the manager puts up with your off-beat attitude because ultimately you make good food, and you keep their small but devoted clientèle coming back. But you can’t be franchised. The manager of the restaurant can’t scale you up without making what you do into a procedure.

So back to Ken’s topic… if you are choosing a systems integrator, you need to decide if you’re buying an accounting system (i.e. something that’s generic to all companies, and not a competitive advantage), or something that is a competitive advantage to you. When you’re automating your core business processes, you must build competitive advantage into it, and it must be innovative. If that’s the case, stay away from larger integrators with miles and miles of red tape and bureaucracy. Go for the “boutique” integrator (somewhere in the 7 to 25 person sized company, under $10 million per year in revenue) that can show you good references. You’re looking for a small group of passionate people. Buzzwords are a warning sign; small companies don’t have time for corporate-speak.

I’m not saying you should use the two guys in their garage. These guys are ok for your basic maintenance tasks, small changes, and local support, but you do want someone who has been around for a few years and has at least a couple of backup engineers they can pull in if there’s a problem. Make sure they have a server, with backups, and all that.

On the other hand, if what you’re automating is very large and very standard, that’s when you want to go with Ken’s approach. If you need to integrate a welding line, paint line, or whatever, there’s nothing new or innovative in that, so you want to lower the risk. You know all the big integration companies can do this, so go and get three bids, and choose the one that’s hungriest for the work. Make sure they have standards and best practices. The reduction in risk is worth it if you don’t need the innovative solution.

You can do a hybrid approach. Identify the parts of your process that could be key competitive advantages if you could find a better way to do it. This is where innovation pays off. Go out and consult with some boutique integrators ahead of time and get them working on those “point solutions”. Then go to the bigger companies to farm out the rest of your automation needs. How’s that for a “best practice”?

The User Interface Makes the Difference, Except in Automation

2 Replies

Start by watching this video about the Aeryon Scout robot (kudos Kareem):

I think what sets this aerial robot apart, as Kareem says, is the intuitive user interface. When I look at the state of automation today, I can see that good user interfaces are typically an after-thought. Custom solutions are sometimes so cobbled together that there isn’t enough bandwidth between one black box and the HMI, or the HMI is just a simple two line text display that ends up saying FAULT 53 (the manual with the list of faults, of course, is stuck inside the door of the panel, and it’s the only thing in the area that isn’t covered in grease because nobody bothers to look at it).

People frequently blame engineers for this mess, which I find a bit silly. Certainly user interfaces are a critical component of any system, but why do you hire an electrical designer to do the electrical design, hire a programmer to write the software, but expect one of these people to magically become a usability expert, which is a field unto itself?

I think there used to be an idea that there was no payback on usability. Certainly if you’re selling something like a VCR, you can only print features on the box (you can’t accurately represent the experience of using it) and people only buy one. However, as items become more social (think iPhone), we’re starting to see great user interfaces create viral marketing for products. I think I first saw this with the TiVo – once you saw what it could do, and how easy it was, you were hooked. Apple’s technology seems to be the same way, and I can see how the Aeryon Scout probably has the same “shock and awe” effect when you demo it.

Where does that leave us with industrial automation interfaces? Automation is always purchased based on a cost-benefit analysis because of the high capital cost. The operators typically don’t participate in the purchasing decision at all. I don’t think effort put into a better user interface is wasted; in fact I’m certain there’s a long term payback. But it’s not a selling feature and it takes more time to do right.

Still, when I programmed a machine recently, it was nice to overhear someone say, “it’s pretty intuitive, isn’t it?” So I guess I’ll keep trying, even if it’s not in my own best interest. Engineers are weird that way.

If you’re interested in making better user interfaces, first I recommend reading The Design of Everyday Things by Donald Norman. I also recommend this video called the least you can do about usability by Steve Krug, author of Don’t Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition.

The Two Kinds of Automation Software

1 Reply

As we all know, there are 10 kinds of people in the world.

For those of you who haven’t read Zen and the Art of Motorcycle Maintenance by Pirsig, he spends at least one chapter at the beginning talking about how we naturally tend to divide things into smaller pieces in an effort to understand them. The novice looks at a motorcycle and sees the visible things, like a seat, handlebars, and wheels, but the expert sees a fuel system, a cooling system, and the suspension. The same thing or system (motorcycle) can be subdivided different ways depending on what we want to do with it.

My tongue-in-cheek title of this post is an acknowledgement of the many ways we can categorize something like Automation Software, but for my purposes today, I’m making two categories: hammers and levels.

A carpenter carries both a hammer and a level, but the two have fundamentally different failure modes. If a hammer stops working, you’ll know it as soon as you try to use it. As long as it hammers in a nail, it doesn’t matter if the hammer is rusty, dirty, scratched or dented, it’s a working hammer. The level, on the other hand, is a measuring instrument. As novices, we assume that it comes from the factory pre-calibrated, and we happily hang our shelf or picture without testing it, but a professional carpenter knows that they have to check their levels for accuracy, or else the level is useless. You could use a level for years, but if one day it stopped being accurate, you probably wouldn’t know. This is a very different situation than the hammer.

Software in general, and automation software in particular, both have similar examples. You never need to “calibrate” the Axis 1 Advanced proximity switch on a machine because if it doesn’t work, the machine won’t make parts (and you’ll know about it instantly, usually via a 2 am phone call). On the other hand, testing data collection logic is surprisingly difficult because the only way to test it is to compare it with a known-good equivalent. Assuming you created this data collection logic to automate away a manual process, the only measuring stick we can check it against is the manual process we’re replacing. Once the system is bought off and we get rid of the paper system, how do you prove that subsequent changes don’t break the data collection system?

It’s tempting to brush off the problem by saying that anyone who makes a subsequent change has to do a full regression test of the system, including the data collection system, but anyone who has worked in a real factory environment knows that this is unlikely to work in practice. Full regression tests are expensive.

In the greater software world, they use automated unit tests. They take the logic being tested and they run it through a series of automated checks to make sure nothing changes. This works well in an environment like PC programming, but is very difficult in practice for PLC programming because (a) you usually need a physical PLC to execute the logic (unless you have some kind of emulator) and (b) the people maintaining the system are likely not familiar with concepts like unit tests, and are likely to undervalue their importance.

This screams for a system-level solution. Take accounting for instance. Double-entry accounting (the use of debits and credits to force every action to be made twice) is deliberately created to help catch manual entry errors. If your debits and credits don’t balance, you know you’ve made a mistake somewhere, and you go back and check your arithmetic.

In the automation world, the solution is to measure every input to the data collection system two ways, analyze and aggregate both separately, and compare the end results. Create a system warning or fault if the results don’t match. For instance, measure the amount of material going into the machine, and measure the amount of material exiting the machine, both as finished product, and scrap. If the input doesn’t match the sum of the outputs over the same time period, you know you have a problem. The system becomes self-checking (a hammer rather than a level).

If you follow this route, you need to take care to avoid some common traps:

  • Don’t re-use logic between the two sides (in fact, try to make them work differently)
  • Try to use different sensors or sensing methods (can we measure the input by speed and duration, and the output by parts and scrap weight?)
  • Record both, so if there is a discrepancy, you can check them against manual measurements

It sounds like more work, but making the system self-checking actually reduces the amount of testing you have to do, so it’s not as bad as you think. Besides, writing code is a lot more fun than testing it. We automate everyone else’s job, why not the boring parts of ours?

SoapBox Snap: Sneak Peek

4 Replies

We crossed a bit of a milestone in the development of SoapBox Snap this weekend… I was able to create an automation solution, connect to the included soft runtime, automatically detect the Phidgets 8/8/8 board I have hooked up to a USB port, and then I wrote a little ladder logic. The ladder logic executes in the runtime, and it’s driving an output on the Phidgets board. Here’s a screenshot:

SoapBox Snap Sneak Peek Screenshot

(Click on the image to enlarge it.)

It’s still a little rough around the edges, but it feels good to have the logic running end-to-end. It starts to feel pretty real at this point.

What is SoapBox Snap?

Snap stands for Snap is Not A PLC. It’s not for industrial control. It’s for home automation enthusiasts, students, and generally anyone who likes to make stuff to add a little automation to their project without necessarily knowing classic computer programming.

Extensibility

Pretty much everything in SoapBox Snap is extensible by 3rd party programmers:

  • Write your own library of ladder logic instructions
  • Plug in a new runtime to replace the “soft” runtime that comes with it
  • Add a new language editor for automation languages other than ladder logic
  • Write a driver for a new I/O device that you want to control from the runtime

Free

SoapBox Snap will be released under the GNU Public License (GPL). That means it’s free. Free-as-in-beer and free-as-in-speech.

The data structure, file format, and communications protocol are encapsulated in a separate library called SoapBox Protocol. It’s going to be concurrently released under the CDDL so you could build SoapBox Snap compatibility into an application without releasing your entire application under the GPL.

Why SoapBox Snap?

As a controls engineer I’ve loved programming machines in ladder logic. It’s intuitive, especially to someone with any amount of electrical background, and it’s interactive, especially if you have a platform that supports online debugging and programming. I love that you don’t have to be a programmer to “get it”.

I think ladder logic could be applied to many domains outside of strict industrial automation, but as I’ve blogged about before, the existing automation equipment vendors have no reason to take ladder logic outside of their cushy niche and bring it to a larger, more mainstream, audience. I don’t want to see the broken innovation of the industrial automation industry hold back other possibly innovative uses of some of these ideas.

Creating a Framework for Innovation

You need two things to get innovation going: an open standard, and an open set of tools. That’s what SoapBox Snap is.

You can try to create an open standard by getting all the stakeholders to sit around a table and creating a specification, but it usually doesn’t work. Exhibit A: the IEC 61131-3 specification. That standard creates little or no interoperability between various automation platforms.

The other way to create an open standard, the one that actually works, is just to write it and open it up under intellectual property protections like the GPL, and give it away for free. The reason you release it under a license like the GPL is because it protects against embrace-and-extend tactics. No single company can take the application, spend a whole bunch of money to add a bunch of features, and then release it as an incompatible version. If they do that, they have to respect the license and release their changes in kind. That lets the standard evolve, but prevents it from becoming proprietary.

SoapBox Snap is for Me

I’ve been looking for something like SoapBox Snap for almost a decade now. A long time ago there was a small group of people who got together to create an open source Linux-based PLC called Puffin PLC. You can still find references to it around the web, but it never came to fruition. I now understand why. Writing a ladder logic editor is full of challenges I didn’t expect. While I hope the same people who were interested in making Puffin PLC will be interested in SoapBox Snap, I’m ultimately building it for me. I’m going to build something cool with it, and it’s going to be fun. 🙂