Category Archives: Automation

Estimating Software Projects

1 Reply

Why is it so hard to estimate software (and control system) projects?

Does this seem familiar? You start out as a grunt in the trenches designing stuff, writing code, and you keep getting a string of projects to work on with insanely low budgets and short timelines, and you think, “I can do better than this.” You decide you want to manage your own projects, and your boss thinks that’s great because (a) he trusts you, and (b) that’s one less thing for him to do. He starts you off with some small project…

The first thing he asks you for is an estimate. You listen to the scope, give it about 10 minutes of thought, write up an estimate and hand it to him:

  • Design: 3 days
  • Programming: 2 weeks
  • Testing: 2 days

So, three weeks. “Let’s just say four weeks,” says your boss. Great!

You start out on your design. It takes about 3 days. Then you start programming, and you get it all working, and you’re on budget. Excellent! Your boss asks you how it’s going, and you say, “I’m on budget! I just have some testing left to do.” Great, let’s have a review with the customer.

After the review with the customer, it’s clear that half of what you did isn’t going to meet their expectations, and the other half is going to get all messed around because of the first half changing. Damn!

“How is this going to affect the budget?” says your boss…

“Well, since we have four weeks, and we’re only two and a half weeks in, we should be ok. I’ll probably have to come in on a weekend.”

So you work your butt off, and the hours are really racking up, but you think you’re going to make it, and all of a sudden, BAM! You have this one annoying problem that you just can’t seem to fix. Maybe it’s a bug in a Windows driver, or a grounding issue, or whatever. It’s something hard, and you burn a whole week trying various solutions trying to solve it. Suddenly you’re overbudget.

After this experience, and a few more like it, you start to get the hint that maybe it’s you. Then you learn there’s an entire industry out there that functions as a support group for people like you: Project Management. You get to go to little two-day project management seminars where they commiserate with you. An instructor gets up and talks about all the common pitfalls of managing a project, and everyone with any experience trying to do it says, “yeah, exactly!”

Then they tell you the solution to this is to impose the tried and true “Project Management Methodology” on it:

  • Initiate
  • Plan
  • Execute
  • Monitor
  • Complete

This gives you a warm fuzzy feeling. The feeling that you’re back in control of your world, and if you’re into control systems like me, then you’re all about controlling your world.

Now the first step includes the estimation step. You do a work breakdown structure, down to small details, and you assign a best case, probable case, and worst case estimates to each task, and you usually use some pseudo-statistics to come up with a weighted sum of the estimates for all tasks.

This process has worked well, but only in certain industries. If you’re building a house, it works well. If you’re building a skyscraper or a bridge, it works well, particularly if the result is similar to something you’ve done before.

It’s odd that most Project Management books use the “send a man to the moon” example. While that project was completed on time (before the end of the decade), and it achieved the performance goal, it went wayyy over budget. If this was a corporate project, it would be a complete failure. It was only considered a success because it was a government job.

If you’re trying to estimate how long it will take to build a house, the estimate starts with a completed design. Then the Project Manager calls up the framers, roofers, drywallers, electricians, plumbers, etc., and gives them a copy of the blueprints. The plumber has a really good idea, based on the number of fixtures, etc., how much it’s going to cost and how long it will take. He does this all the time.

Software is a different beast. It’s like asking you to estimate the architect’s time when he drew up the blueprints of the house. The construction of the house is analogous to the compiling step in software development. In software, we’ve completely automated that step. We can go from design to prototype in a matter of seconds. What we’re estimating in software development is the design effort, not the construction effort. Unfortunately, before the design is done, we haven’t even figured out what the problems are yet, let alone the solutions.

Computer programming is almost always novel. Repetition is the ultimate sin. If you find repetition, you refactor it out into a re-usable library or module. Each project brings new platforms and technologies, ever higher levels of abstraction, and a novel problem, so we have no baseline to estimate from.

In fact, the easier it is to estimate a project, the less important that project really is. If your job is easy to estimate, it’s probably repetitive enough that it’ll soon be automated away.

Finally Getting an Arduino

7 Replies

I cruised through January in a kind of sleep deprived stupor (we just had our second child this December). Things are finally swinging back to normal, and I’m getting my geek back on.

I’ve been looking for a less expensive way to do discrete (or analog) I/O over WiFi for homebrew projects. I want something that can be compatible with SoapBox Snap (the open source ladder logic editor/runtime I’ve been working on), so I’ll need to be able to write a communication driver in C#. I’ve been rather frustrated by the options:

  • National Control Devices WiFi Relay Boards – but they start at $250 and go up from there.
  • Insteon – perfect for home automation and more reliable than X10, but their software license agreement for their development kit is extremely unfriendly to open source.
  • Belkin and other manufacturers have created some wireless USB hubs, but I can’t seem to find anyone who has them in stock, most have been discontinued, and the cost is still prohibitive ($100+) especially when you figure you still have to buy some Phidgets boards on top of that.

Then I finally decided this was my best choice: the YellowJacket WiFi Arduino. Arduino is a family of open-source hardware designs for microcontroller boards. You can buy add-on cards for them (called shields), but you can also purchase custom designs with specific features, like this one with built-in 802.11b WiFi.

The price is right ($55), it’s very open source friendly, and since I get to program the microcontroller end, I should have no problem writing a driver for it in C#. Unfortunately it’s on back order, but I expect to get it in a couple of weeks. I’ll post more after I’ve played with it a bit.

I must admit there’s one other honourable mention, though it was a bit too much of a hack for me. There are these cheap routers you can get called La Fonera (made by FON). It turns out that they have 3 or 4 unused general purpose TTL level I/O on the circuit board, and if you install a copy of the open source DD-WRT firmware on it, it lets you control those I/O using the command line prompt if you telnet or SSH into the router. Perfect, and you can pick these things up really cheap on eBay. Unfortunately I wanted something just a little more off-the-shelf than that. (I do have a copy of DD-WRT running on my old Linksys router and I’m using it as a wireless bridge to extend the range of the wireless in my house, so I did give it some serious consideration.)

Control System Security and 2010

Leave a reply

Looking back at the year 2010, there was one really interesting and important happening in the world of industrial control system security: Stuxnet.

There’s a lot of speculation about this computer worm, but let’s just look at the facts:

  • It required substantially more resources to create than a typical computer worm (some estimates put it around $1,000,000, if you figure 5 person-years and the cost to employ specialized programmers)
  • It targets Siemens WinCC software, so that it can infect Step 7 PLCs
  • It looks like it was specifically targeted at a single facility (based on the fact that it was targeting a specific PLC, and only specific brands of VFDs)
  • It was designed to do real physical damage to equipment
  • It was designed to propagate via USB memory sticks to make it more likely to spread inside industrial settings, and even updated itself in a peer-to-peer manner (new versions brought in from the outside could update copies already inside a secure network)

If your average computer worm is the weapon-equivalent a hatchet, Stuxnet is a sniper rifle. There is speculation that the intended target was either the Bushehr Nuclear Power Plant or the Natanz Nuclear Facility, both in Iran, but what is known is that it has spread far and wide in other industrial networks. It’s been called the world’s first cyber super weapon.

What’s clear is that our industry’s relative ignorance when it comes to computer security has to end. Stuxnet proved the worst case, which is that a proprietary embedded PLC can successfully be the target of a computer worm.

I’ve been watching as more and more old-school vendors include Windows CE based devices as HMI components in their systems (like the PanelView Plus from Allen-Bradley). These are susceptible to the same kinds of threats that can infect Microsoft-based smartphones, and it takes a lot less than $1,000,000 to write one of those. It’s the kind some kid can put together in his basement for fun.

I’ve also seen (and even been pushing) a trend towards PC-based control. It’s nothing new, and I’ve seen PC-based control solutions out there for almost 10 years now, but it’s the networking that makes them vulnerable. In one facility about five years ago, I saw a PC-based control system completely taken down by a regular old computer worm. There were two mitigating causes in that case… first, the control system was on the same network as the main office network (the virus was brought in by an employee’s laptop that they connected at home), and secondly the vendor of the control software prohibited the customer from installing anti-virus software on the control system PC because they said it would void the warranty. I rarely see these same mistakes in new installations, but it does happen.

A couple of years ago I found a computer virus on an industrial PC with a VB6 application used as an HMI for a PLC/2. The PC in question was never connected to any network! The virus found its way to this computer over floppy disks and USB memory sticks.

Now if your facility is juicy enough that someone would spend $1,000,000 to take a shot at it, then you need specialized help. Stuxnet is a boon to security consultants because now they have a dramatic story to wave in the face of clients. On the other hand, most facilities need some kind of basic security measures.

  • Separate your industrial and office networks (if you need to move data from one to the other, then have a secure machine that can sit on both networks)
  • Make sure all machines automatically update their Windows Updates and their anti-virus definitions automatically, even if they’re on the industrial network
  • Change the default passwords on all devices and servers (including SQL Server’s SA password!)
  • Use different technologies in different layers (does your office network use Cisco managed switches? Consider using industrial managed switches from a different vendor in your industrial network)

Are you an integrator looking to expand your lines of business? Hire a computer security consultant and have them go knocking on the doors of your biggest customers with the Stuxnet story. You should be able to sell them a security assessment, and an action plan. Given the current security landscape, you’ll be doing them a big favour.

Intro to Mercurial (Hg) Version Control for the Automation Professional

2 Replies

There’s a tool in the PC programming world that nobody would live without, but almost nobody in the PLC or Automation World uses: version control systems. Even most PC programmers shun version control at first, until someone demonstrates it for them, and then they’re hooked. Version control is great for team collaboration, but most individual programmers working on a project use it just for themselves. It’s that good. I decided since most of you are in the automation industry, I’d give you a brief introduction to my favourite version control system: Mercurial.

It’s called “Mercurial” and always pronounced that way, but it’s frequently referred to by the acronym “Hg”, referring to the element “Mercury” in the periodic table.

Mercurial has a lot of advantages:

  • It’s free. So are all of the best version control systems actually. Did I mention it’s the favourite tool of PC programmers? Did I mention PC programmers don’t like to pay for stuff? They write their own tools and release them online as open source.
  • It’s distributed. There are currently two “distributed version control systems”, or DVCS, vying for supremacy: Hg and Git. Distributed means you don’t need a connection to the server all the time, so you can work offline. This is great if you work from a laptop with limited connectivity, which is why I think it’s perfect for automation professionals. Both Hg and Git are great. The best comparison I’ve heard is that Hg is like James Bond and Git is like MacGyver. I’ll let you interpret that…
  • It has great tools. By default it runs from the command line, but you never have to do that. I’ll show you TortoiseHg, a popular Windows shell that lets you manage your versioned files inside of a normal Windows Explorer window. Hg also sports integration into popular IDEs.

I’ll assume you’ve downloaded TortoiseHg and installed it. It’s small and also free. In fact, it comes bundled with Mercurial, so you only have to install one program.

Once that’s done, follow me…

First, I created a new Folder on my desktop called My New Repository:

Now, right click on the folder. You’ll notice that TortoiseHg has added a new submenu to your context menu:

In the new TortoiseHg submenu, click on “Create Repository Here”. What’s a Repository? It’s just Hg nomenclature for a folder on your computer (or on a network drive) that’s version controlled. When you try to create a new repository, it asks you this question:

Just click the Create button. That does two things. It creates a sub-folder in the folder you right-clicked on called “.hg”. It also creates a “.hgignore” file. Ignore the .hgignore file for now. You can use it to specify certain patterns of files that you don’t want to track. You don’t need it to start with.

The .hg folder is where Mercurial stores all of its version history (including the version history of all files in this repository, including all files in all subdirectories). This is a particularly nice feature about Mercurial… if you want to “un-version” a directory, you just delete the .hg folder. If you want to make a copy of a repository, you just copy the entire repository folder, including the .hg subdirectory. That means you’ll get the entire history of all the files. You can zip it up and send it to a friend.

Now here’s the cool part… after you send it to your friend, he can make a change to the files while you make changes to your copy, you can both commit your changes to your local copies, and you can merge the changes together later. (The merge is very smart, and actually does a line-by-line merge in the case of text files, CSV files, etc., which works really well for PC programming. Unfortunately if your files use a proprietary binary format, like Excel or a PLC program, Mercurial can’t merge them, but will at least track versions for you. If the vendor provides a proprietary merge tool, you can configure Mercurial to open that tool to merge the two files.)

Let’s try an example. I want to start designing a new automation cell, so I’ll just sketch out some rough ideas in Notepad:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder

I save it as a file called “Line 1.txt” in my repository folder. At this point I’ve made changes to the files in my repository (by adding a file) but I haven’t “committed” those changes. A commit is like a light-weight restore point. You can always roll back to any commit point in the entire history of the repository, or roll forward to any commit point. (You can also “back out” a single commit even if it was several changes ago, which is a very cool feature.) It’s a good idea to commit often.

To commit using TortoiseHg, just right click anywhere in your repository window and click the “Hg Commit…” menu item in the context menu. You’ll see a screen like this:

It looks a bit weird, but it’s showing you all the changes you’ve made since your “starting point”. Since this is a brand new repository, your starting point is just an empty directory. Once you complete this commit, you’ll have created a new starting point, and it will track all changes after the commit. However, you can move your starting point back and forth to any commit point by using the Update command (which I’ll show you later).

The two files in the Commit dialog box show up with question marks beside them. That means it knows that it hasn’t seen these files before, but it’s not sure if you want to include them in the repository. In this case you want to include both (notice that the .hgignore file is also a versioned file in the repository… that’s just how it works). Right click on each one and click the Add item from the context menu. You’ll notice that it changes the question mark to an “A”. That means the file is being added to the repository during this commit.

In the box at the top, you have to enter some description of the change you’re making. In this case, I’ll say “Adding initial Line 1 layout”. Now click the Commit button in the upper left corner. That’s it, the file is now committed in the repository. Close the commit window.

Now go back to your repository window in Windows Explorer. You’ll notice that they now have green checkmark icons next to them (if you’re using Vista or Windows 7, sometimes you have to go into or out of the directory, come back in, and press F5 to see it update):

The green checkmark means the file is exactly the same as your starting point. Now let’s try editing it. I’ll open it and add a Zone 4 to the conveyor:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
    - Zone 4
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder

The icon in Windows Explorer for my “Line 1.txt” file immediately changed from a green checkmark to a red exclamation point. This means it’s a tracked file and that file no longer matches the starting point:

Notice that it’s actually comparing the contents of the file, because if you go back into the file and remove the line for Zone 4, it will eventually change the icon back to a green checkmark!

Now that we’ve made a change, let’s commit that. Right click anywhere in the repository window again and click “Hg Commit…”:

Now it’s showing us that “Line 1.txt” has been Modified (see the M beside it) and it even shows us a snapshot of the change. The box in the bottom right corner shows us that we added a line for Zone 4, and even shows us a few lines before and after so we can see where we added it. This is enough information for Mercurial to track this change even if we applied this change in a different order than other subsequent changes. Let’s finish this commit, and then I’ll give you an example. Just enter a description of the change (“Added Zone 4 to Conveyor”) and commit it, then close the commit window.

Now right-click in the repository window and click on Hg Repository Explorer in the context menu:

This is showing us the entire history of this repository. I’ve highlighted the last commit, so it’s showing a list of files that were affected by that commit, and since I selected the file on the left, it shows the summary of the changes for that file on the right.

Now for some magic. We can go back to before the last commit. You do this by right-clicking on the bottom revision (that says “Adding initial Line 1 layout”) and selecting “Update…” from the context menu. You’ll get a confirmation popup, so just click the Update button on that. Now the bottom revision line is highlighted in the repository explorer meaning the old version has become your “starting point”. Now go and open the “Line 1.txt” file. You’ll notice that Zone 4 has been removed from the Conveyor (don’t worry if the icons don’t keep up on Vista or Win7, everything is working fine behind the scenes).

Let’s assume that after the first commit, I gave a copy of the repository to someone, (so they have a copy without Zone 4), and they made a change to the same file. Maybe they added some detail to the Nut Feeder section:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder
    - 120VAC, 6A

Then they committed their change. Now, how do their changes make it back into your repository? That’s by using a feature called Synchronize. It’s pretty simple if you have a copy of both on your computer, or if each of you have a copy, and you also have a “master” copy of the repository on the server, and you can each see the copy on the server. What happens is they “Push” their changes to the server copy, and then you “Pull” their change over to your copy. (Too much detail for this blog post, so I’ll leave that to you as an easy homework assignment). What you’ll end up with, when you look at the repository explorer, is something like this:

You can clearly see that we have a branch. We both made our changes to the initial commit, so now we’ve forked it. This is OK. We just have to do a merge. In a distributed version control system, merges are normal and fast. (They’re fast because it does all the merge logic locally, which is faster than sending everything to a central server).

You can see that we’re still on the version that’s bolded (“Added Zone 4 to Conveory”). The newer version, on top, is the one with the Nut Feeder change from our friend. In order to merge that change with ours, just right click on their version and click “Merge With…” from the context menu. That will give you a pop-up. It should be telling you, in a long winded fashion, that you’re merging the “other” version into your “local” version. That’s what you always want. Click Merge. It will give you another box with the result of the merge, and in this case it was successful because there were no conflicts. Now click Commit. This actually creates a new version in your repository with both changes, and then updates your local copy to that merged version. Now take a look at the “Line 1.txt” file:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
    - Zone 4
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder
    - 120VAC, 6A

It has both changes, cleanly merged into a single file. Cool, right!?

What’s the catch? Well, if the two changes are too close together, it opens a merge tool where it shows you the original version before either change, the file with one change applied, the file with the other change applied, and then a workspace at the bottom where you can choose what you want to do (apply one, the other, both, none, or custom edit it yourself). That can seem tedious, but it happens rarely if the people on your project are working on separate areas most of the time, and the answer of how to merge them is usually pretty obvious. Sometimes you actually have to pick up the phone and ask them what they were doing in that area. Since the alternative is one person overwriting someone else’s changes wholesale, this is clearly better.

Mercurial has a ton of other cool features. You can name your branches different names. For example, I keep a “Release” branch that’s very close to the production code where I can make “emergency” fixes and deploy them quickly, and then I have a “Development” branch where I do major changes that take time to stabilize. I continuously merge the Release branch into the Development branch during development, so that all bug fixes make it into the new major version, but the unstable code in the Development branch doesn’t interfere with the production code until it’s ready. I colour-code these Red and Blue respectively so you can easily see the difference in the repository explorer.

I use the .hgignore file to ignore my active configuration files (like settings.ini, for example). That means I can have my release and development branches in two different folders on my computer, and each one has a different configuration file (for connecting to different databases, or using different file folders for test data). Mercurial doesn’t try to add or merge them.

It even has the ability to do “Push” and “Pull” operations over HTTP, or email. It has a built-in HTTP server so you can turn your computer into an Hg server, and your team can Push or Pull updates to or from your repository.

I hope this is enough to whet your appetite. If you have questions, please email me. Also, you can check out this more in-depth tutorial, though it focuses on the command-line version: hginit.

Will TwinCAT 3 be Accepted by Automation Programmers?

14 Replies

Note that this is an old article and I now have more up-to-date TwinCAT 3 Reviews and now a TwinCAT 3 Tutorial.

In the world of programming there are a lot of PC programmers and comparatively few PLC programmers, but I inhabit a smaller niche. I’m a PLC and a PC programmer. This is a dangerous combination.

If you come from the world of PLC programming, like I did, then you start out writing PC programs that are pretty reliable, but they don’t scale well. I came from an electrical background and I adhered to the Big Design Up Front (BDUF) methodology. The cost of design changes late in the project are so expensive that BDUF is the economical model.

If you come from the world of PC programming, you probably eschew BDUF for the more popular “agile” and/or XP methodologies. If you follow agile principles, your goal is to get minimal working software in front of the customer as soon as possible, and as often as possible, and your keep doing this until you run out of budget. As yet there are no studies that prove Agile is more economical, but it’s generally accepted to be more sane. That’s because of the realization that the customer just doesn’t know what they want until they see what they don’t want.

It would be very difficult to apply agile principles to hardware design, and trying to apply BDUF (and the “waterfall” model) to software design caused the backlash that became Agile.

Being both a PLC and a PC programmer, I sometimes feel caught between these two worlds. People with electrical backgrounds tend to dislike the extra complexity that comes from the layers and layers of abstraction used in PC programming. Diving into a typical “line of business” application today means you’ll need to understand a dizzying array of abstract terminology like “Model”, “View”, “Domain”, “Presenter”, “Controller”, “Factory”, “Decorator”, “Strategy”, “Singleton”, “Repository”, or “Unit Of Work”. People from a PC programming background, however, tend to abhor the redundancy of PLC programs, not to mention the lack of good Separation of Concerns (and for that matter, source control, but I digress).

These two worlds exist separately, but for the same reason: programs are for communicating to other programmers as much as they’re for communicating to machines. The difference is that the reader, in the case of a PLC program, is likely to be someone with only an electrical background. Ladder diagram is the “lingua franca” of the electrical world. Both electricians and electrical engineers can understand it. This includes the guy who happens to be on the night shift at 2 am when your code stops working, and he can understand it well enough to get the machine running again, saving the company thousands of dollars per minute. On the other hand, PC programs are only ever read by other PC programmers.

I’m not really sure how unique my situation is. I’ve had two very different experiences working for two different Control System Integrators. At Patti Engineering, almost every technical employee had an electrical background but were also proficient in PLC, PC, and SQL Server database programming. On the other hand, at JMP Engineering, very few of us could do both, the rest specialized in one side or the other. In fact, I got the feeling that the pure PC programmers believed PLC programming was beneath them, and the people with the electrical backgrounds seemed to think PC programming was boring. As one of the few people who’ve tried both, I can assure you that both of these technical fields are fascinating and challenging. I also believe that innovation happens on the boundaries of well established disciplines, where two fields collide. If I’m right, then both my former employers are well positioned to cash in on the upcoming fusion of data and automation technologies.

TwinCAT 3

I’ve been watching Beckhoff for a while because they’re sitting on an interesting intersection point.

On the one side, they have a huge selection of reasonably priced I/O and drive hardware covering pretty much every fieldbus you’d ever want to connect to. All of their communication technologies are built around EtherCAT, an industrial fieldbus of their own invention that then became an open standard. EtherCAT, for those who haven’t seen it, has two amazing properties: it’s extremely fast, compared with any other fieldbus, and it’s inexpensive, both for the cabling and the chip each device needs to embed for connectivity. It’s faster, better, and cheaper. When that happens, it’s pretty clear the old technologies are going to be obsolete.

On the other side, they’re a PC-based controls company. Their PLC and motion controllers are real-time industrial controllers, but you can run them on commodity PC hardware. As long as PCs continue to become more powerful, Beckhoff’s hardware gets faster, and they get those massive performance boosts for free. Not only that, but they get all the benefits of running their PLC on the same machine as the HMI, or other PC-based services like a local database. As more and more automation cells need industrial PCs anyway, integrators who can deliver a solution that combines the various automation modules on a single industrial PC will be more competitive.

Next year Beckhoff is planning to release TwinCAT 3, a serious upgrade from their existing TwinCAT 2.11. The biggest news (next to support for multiple cores) is that the IDE (integrated development environment) is going to be built around Microsoft’s Visual Studio IDE. That’s a pretty big nod to the PC programmers… yes you can still write in all the IEC-61131-3 languages, like ladder, function block, etc., but you can also write code in C/C++ that gets compiled down and run in the real-time engine.

Though it hasn’t been hyped as much, I’m pretty excited that you can have a single project (technically it’s called a “solution”) that includes both automation programming, and programming in .NET languages like C# or VB.Net. While you can’t write real-time code in the .NET languages, you can communicate between the .NET and real-time parts of your system over the free ADS communication protocol that TwinCAT uses internally. That means your system can now take advantage of tons of functionality in the .NET framework, not to mention the huge amount of 3rd party libraries that can be pulled in. In fact, did you know that Visual Studio has a Code Generation Engine built in? It would be pretty cool to auto-generate automation code, like ladder logic, from templates. You’d get the readability of ladder logic without the tedious copy/paste/search/replace. (Plus, Visual Studio has integrated source control, but I digress…)

Will anyone take advantage?

With such a split between PC and PLC programmers, exactly who is Beckhoff targeting with TwinCAT 3? They’ve always been winners with the OEM market, where the extra learning curve can be offset by lower commodity hardware prices in the long term. I think TwinCAT 3 is going to be a huge win in the OEM market, but I really can’t say where it’s going to land as far as integrators are concerned. Similar to OEMs, I think it’s a good fit for integrators that are product focused because the potential for re-use pays for your ramp-up time quickly.

It’s definitely a good fit for my projects. I’m interested to see how it turns out.

Overengineering

1 Reply

“Overengineering” is a word that gets thrown around a lot. It’s used in a negative connotation, but I have a hard time defining it.

It’s not the same as Premature Optimization. That’s when you add complexity in order to improve performance, at the expense of readability, but the payoff isn’t worth the cost.

If “to engineer” is synonymous with “to design”, then overengineering is spending too much time designing, and not enough time… what? Implementing?

Lets say you and I need to travel across the continent. You despise overengineering, so you set off on foot immediately, gaining a head start. I go and buy some rollerblades, and easily pass you before the end of the day. Seeing me whiz past, you head to the nearest sporting goods store and buy a ten-speed. You overtake me not long after breakfast on the second day. “Hmm,” I think. I don’t have much money, but I rollerblade on over to a junk yard and get an old beater car. It doesn’t run though. I do some trouble-shooting… the electrical system is fine, and we have spark, but we’re just not getting ignition. I might be able to fix it, and I might not. Should I go and buy a faster bike than yours, and try to catch up, or should I take my chance and see if I can fix this car? I’m pretty sure I can fix it, and if I can, I can easily win, but if I can’t, I’m giving up the lower-risk but lower probability chance of winning by getting a bike.

It’s this last type of choice that we’re faced with as engineers. You have a project with an 8-week timespan. We estimated that it will take 10 weeks at 50 hours per week using standard practices, so the project manager just wants everyone to work 60+ hour weeks using the same practices because from their point of view, that’s the “safe” bet. As an engineer, you might be able to spend 3 weeks building something with a 90% chance of making you 2 times more efficient at building this type of project: 3 weeks spent building the tool, and then it would only take 5 weeks to complete the project, so you’re done in 8 weeks. Not only that, but then you’ve got a tool you can re-use the next time.

If every time we had this choice, we chose to make the tool first, then on average we’ll end up much further ahead. Every time we succeed (90% of the time), we’ll greatly improve our capabilities. We’ll out-innovate the competition, with lower costs and faster time to market. However, a manager is much more likely not to build the tool because they can’t tolerate the risk. The larger the company, the worse this is, and the typical excuse leveled at the “tool” solution is that it’s “overengineering.”

Imagine we’re back in the cross-continent scenario, and I’ve decided to fix the car. Two days later I’ve got car parts all over the ground, and I haven’t moved an inch. Meanwhile, you’re a hundred miles away from me on your bike. Who’s winning the race? You can clearly point to your progress… it’s progress that anyone can clearly see. I, on the other hand, can only show someone a car that’s seemingly in worse shape than it started in, plus my inability to move over the last few days. The pressure starts to mount. It’s surprising how quickly people will start to criticize the solution they don’t understand. They’ll call me a fool for even attempting it, and applaud you on your straightforward approach.

Of course you realize that if I can get the car working, the game’s over. By the time you see me pass you, it’ll be too late to pull the same trick yourself. I’ll already be travelling fast enough that you can’t catch me. If there’s a 90% chance I can fix the car, I’d say that’s the logical choice.

So is fixing the car “overengineering”? If the race was from here to the corner, then yes, I’d say so. The effort needs to be matched to the payback. Even if the race were from here to the next town, it wouldn’t give you a payback. But what if we were going to race from here to the next town once every day for the rest of the year? Wouldn’t it make sense to spend the first week getting myself a car, and then win the next 51 weeks of races?

In business, we’re in it for the long haul. It makes sense to spend time making ourselves more efficient. Why, then, do so many companies have systems that encourage drastic short term decision making at the expense of long term efficiencies and profit? How do we allow for the reasonable risk of failure in order to collect the substantial reward of innovation?

You start by finding people who can see the inefficiencies — the ones who can see what could easily be improved, streamlined, and automated. Then you need to take those people out of the environment where every minute they’re being pushed for another inch of progress. Accept that failure is a possible outcome once in a while. Yes, there’s risk, but there are also rewards. One doesn’t come without the other.

The Controls Engineer

2 Replies

An unbelievable buzz at quarter past eight
disturbed my deep thoughts; It’s my phone on vibrate.

It crawls ‘cross the desk, two inches or more
If I leave it, I wonder, will it fall on the floor?

I answer it finally, it’s a privilege you see
to have this fine gilded leash fastened to me.

It turns out it’s Mike in the maintenance shack
He says they’ve been fighting the dispenser out back.

“No problem,” I say, “I’ll come have a look-see”
then closing my phone I gulp back my coffee.

What do I need? My laptop, for sure,
a null modem cable, three adapters, or four?

I’ve got TWO gender benders, that should be enough.
I used to have more; I keep losing this stuff.

I glance at my tool kit, haven’t used it since June-and
I won’t use it again since we got this union.

My battery gives me ’bout ten minutes power
I’ll take my adapter; driving back here’s an hour.

Then out to my car, on my way to Plant 2
they phone me again, three text messages too.

I’m over there fast but no parking in sight.
The overflow lot’s one block down on the right.

Up to the entrance and in through the door,
Remember to sign at the desk, nine-oh-four.

My old ID badge doesn’t work with this scanner
I wonder when she will be back from the can, or

should I just get someone else to come get me?
Mike doesn’t answer, I try Mark… how ’bout Jenny?

“Hi Jenny… never mind, the receptionist’s back.”
The door latch, it closes behind me, click-clack.

Out on the floor passing blue and white panels
Watch out for things painted caution-tape-yellow.

On the right is that cell with the new network NIC,
It didn’t work well with that 5/05 SLC.

To the left is the line I commissioned in May.
It’s sat idle so far; warranty’s up next Friday.

Two more aisles down this way, a left then a right.
Hey! Now I see the dispenser in sight.

“Good morning, Mike,” I said, “How can I help?”
Mike says, “Don’t worry mate, it was just a loose belt.”

When to use a Sealed Coil vs. a Latch/Unlatch?

3 Replies

I just realized something I didn’t learn until at least a year into programming PLCs, and thought it would be a great thing to share for newer ladder logic programmers: when should you use a sealed-in coil vs. a latch/unlatch?

On the surface of it, a latch/unlatch instruction is sometimes frowned upon by experienced programmers because it’s correlated with bad programming form: that is, modifying program state in more than one location in the program. If you have one memory bit that you’re latching and unlatching all over the place, it really hinders readability, and I pity the fool that has to troubleshoot that code. Of course, most PLCs let you use the same memory bit in a coil instruction as much as you want, and that’s equally bad form, so I don’t take too strict of a stance on this. If you are going to use latch/unlatch instructions, make sure you only use one of each (for a given memory bit), and keep them very close together (preferably on adjacent rungs, or even in different branches of the same rung). Don’t make the user scroll, or worse yet, do a cross reference.

As you can imagine, if you’re going to use a Latch/Unlatch instruction and keep them very close together, it’s trivial to convert that to a rung with a sealed in coil, so what, if anything is the difference? Why have two sets of instructions that do the same thing?

It turns out (depending on the PLC hardware you’re using) that they act differently. On Allen-Bradley hardware, at least, an OTE instruction (coil) will always be reset (cleared to off) during the pre-scan. The pre-scan happens any time you restart the program, which is most importantly after a loss of power. If you’re using a sealed in coil to remember you have a pallet present in a zone, you’ll be in for a big surprise when you cycle power. All your zones will be unblocked, and you could end up with a bunch of crashes! On the other hand, OTL and OTU instructions don’t do anything during a pre-scan, so the state remains the same as it was before the power was removed.

For that reason, a latch/unlatch is a great indication of long term program state. If you have to track physical state about the real world, use a latch/unlatch instruction.

On the other hand, a sealed-in coil is a great way to implement a motion command (e.g. “attempting to advance axis A”). In that case you want your motion command to reset if the power drops out.

I hope that clears it up a bit. I always tried to avoid all latch/unlatch instructions until I understood these concepts.


Narrowing the Problem Domain

4 Replies

One of the ongoing tasks in industrial automation is troubleshooting. It’s not glamorous, and it’s a quadrant one activity, but it’s necessary. Like all quadrant one activities, the goals is to get it done as fast as possible so you can get back to quadrant two.

Troubleshooting is a process of narrowing the problem domain. The problem domain is all the possible things that could be causing the problem. Let’s say you have a problem getting your computer on the network. The problem can be any one of these things:

  • Physical network cable
  • Networks switch(es)
  • Network card
  • Software driver
  • etc.

In order to troubleshoot as quickly as possible, you want to eliminate possibilities fast (or at least determine which ones are more likely and which are unlikely). If you don’t have much experience, your best bet is to figure where the middle point is, then isolate the two halves and determine which half seems to be working right and which isn’t. This is guaranteed to reduce the problem domain by 50% (assuming there’s only one failure…). So, in the network problem, the physical cable is kind of in the middle. If you unplug it from the back of the computer and plug it into your laptop, can the laptop get on the internet? If yes, the problem’s in your computer, otherwise, it’s upstream. Rinse and repeat.

As you start to gain experience, you start to get faster because you can start to assign relative probabilities of failure to each component. Maybe you’ve had a rash of bad network cards recently, so you might start by checking that.

In industrial automation, I’ve seen a pattern that pops up again and again that helps me narrow the problem domain, so I thought I’d share. Consider this scenario: someone comes to you with a problem: “the machine works fine for a long time, and then it starts throwing fault XYZ (motion timeout), and then after ten minutes of clearing faults, it’s working again.” These annoying intermittent problems can be a real pain, because it’s sometimes hard to reproduce the problem, and it’s hard to know if you’ve fixed it.

However, if you ask yourself one more question, you can easily narrow it down. “Is the sensor that detects the motion complete condition a discrete or analog sensor?” If it’s a discrete sensor, the chance that the problem is in the logic is almost nil. I know our first temptation is always to break out the laptop, and a lot of people have this unrealistic expectation that we can fix stuff like this with a few timers here or there, but that’s not going to help. If you have discrete logic that runs perfectly for a long time and then suddenly has problems, it’s unlikely there’s a problem in the logic. There’s a 99% certainty that it’s a physical problem. Start looking for physical abnormalities. Does the sensor sense material or a part? If yes, is the sensor position sensitive to normal fluctuations in the material specifications? Is the sensor affected by ambient light? Is the sensor mount loose? Is the air pressure marginal? Is the axis slowing down due to wear?

The old adage, “when all you have is a hammer, every problem is a nail”, is just as true when the only tool you have is a laptop. Don’t break out the laptop when all you need is a wrench.

Book Review: The Lights in the Tunnel

2 Replies

I was paging through the Amazon store on my Kindle when I came across a book that caught my eye: The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future (Volume 1)

It’s not every day you come across a book about Automation, and for $6, I figured, “what the heck?”

The author, Martin Ford, is a Computer Engineer from California. To summarize, he’s basically saying the following:

  • Within 80 years, we will have created machines that will displace significantly more than half of the current workforce

This is a topic that interests me. Not only do I have a career in automation, but I’ve previously wondered about exactly the same question that Ford poses. What happens if we create machines advanced enough that a certain segment of the population will become permanently unemployed?

The title of the book comes from Ford’s “Tunnel Analogy”. He tries to model the economy as a tunnel of lights, with each light representing a person, its brightness indicating its wealth, and the tunnel is lined with other patches of light: businesses. The lights float around interacting with the businesses. Some businesses grow larger and stronger while others shrink and die off, but ultimately the brightness of the tunnel (the sum of the lights) appears to be increasing.

I found the analogy to be a bit odd myself. Actually, I wasn’t quite sure why an analogy was necessary. We’re all pretty familiar with how the free market works. If you don’t get it, I don’t think the tunnel analogy is going to help you. In fact, one excerpt from his description of the tunnel makes me wonder if Ford himself even “gets” the concept of how the economy works:

As we continue to watch the lights, we can now see that they are attracted to the various panels. We watch as thousands of lights steam toward a large automaker’s panels, softly make contact and then bounce back toward the center of the tunnel. As the lights touch the panel, we notice that they dim slightly while the panel itself pulses with new energy. New cars have been purchased, and a transfer of wealth has taken place.

That particular statement irked me during the rest of the book. That’s not a good illustration of a free market; that’s an illustration of a feudal system. In a free market, we take part in mutually beneficial transactions. The automaker has a surplus of cars and wants to exchange them for other goods that it values more, and the consumer needs a car and wants to exchange his/her goods (or promise of debt) in exchange for the car. When the transaction takes place, presumably the automaker has converted a car into something they wanted more than the car, and the consumer has converted monetary instruments into something they wanted more: a car. Both the automaker and the consumer should shine brighter as a result of the transaction.

Ford has confused money with wealth, and that’s pretty dangerous. As Paul Graham points out in his excellent essay on wealth:

Money Is Not Wealth

If you want to create wealth, it will help to understand what it is. Wealth is not the same thing as money. Wealth is as old as human history. Far older, in fact; ants have wealth. Money is a comparatively recent invention.

Wealth is the fundamental thing. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn’t need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn’t matter how much money you had.

There are actually two ways to create wealth. First, you can make it yourself (grow your own food, fix your house, paint a picture, etc.), or secondly you can trade something you value less for something you value more. In fact, most of us combine these two methods: we go to work and create something that someone else wants so we can trade it for stuff that we want (food, cars, houses, etc.).

Later in the book, Ford makes a distinction between labor-intensive and capital-intensive industries. He uses YouTube as an example of a capital-intensive business because they were purchased (by Google) for $1.65B and they don’t have very many employees. I can’t believe he’s using YouTube as an example of a capital-intensive industry. The new crop of online companies are extremely low-overhead endeavors. Facebook was started in a dorm room. Again, Ford seems to miss the fact that money is not equal to wealth. Google didn’t buy YouTube for the capital, they bought their audience. Google’s bread and butter is online advertising, so they purchased YouTube because users are worth more to Google than they are to the shareholders of YouTube that sold out. Wealth was created during the transaction because all parties feel they have something more than they had to start.

Back to Ford’s premise for a moment: is it possible that we could create machines advanced enough that the average person would have no place in a future economy? I don’t find it hard to believe that we could eventually create machines capable of doing most of the work that we do right now. We’ve certainly already created machines that do most of the work that the population did only decades ago. The question is, can we get to the point where the average person has no value to add?

Let’s continue Ford’s thought experiment for a moment. You and I and half the population is now out of work and nobody will hire us. Presumably the applicable constitutional elements are in place so we’re still “free”. What do we do? Well, I don’t know about you, but if I had no job and I was surrounded by a bunch of other people with no job, I’d be out foraging for food. When I found some, I’d probably trade a bit of it to someone who could sew that might patch up my shirt. If I had a bit of surplus, I’d probably plant a few extra seeds the next spring and get a decent harvest to get me through the next winter.

I’m not trying to be sarcastic here. I’m trying to point out the obvious flaw in the idea that a large percentage of the population couldn’t participate in the economy. If that were the case, the large part of the population would, out of necessity, form their own economy. In fact, if we’re still playing in Ford’s dreamland here, where technology is so advanced that machines can think and perhaps even nanotechnology is real, I’d probably hang around the local dump and forage for a bit of technology there. The treasures I’d find there would probably put me in more luxury than I currently have in 2010.

So, if you take the thought experiment to the extreme, it breaks down. Given a free society divided into haves and have-nots, where the haves don’t have any use for the have-nots, then what you really have is two separate and distinct societies, each with its own bustling economy. Whether or not there is trade between those two economies, one thing is certain: almost everyone still has a job.

Of course, it’s not like we’re going to wake up tomorrow and technology will suddenly throw us all out of our jobs. The shift in technology will happen gradually over time. As technology improves, people will need to adapt (as we do every day). As I’ve said before, I think a major shift away from the mass consumption of identical items is already underway. As the supply of generic goods goes up, our perceived value of them goes down.

Ford doesn’t seem to participate in automation on a daily basis, so I think he lacks the experience of what automation really does. Automation drives down the cost, but it also increases the supply and reduces the novelty at the same time. Automated manufacturing makes products less valuable but the juxtaposition makes people more valuable.

There’s a company out there called Best Made Co. that sells $200 hand-made axes. There’s a three week waiting time. That’s a feature actually: it’s so valuable to people that there’s a three week lead time. It’s made by hand. It’s made by hand by people who are passionate about axes. Feature? I think so.

In Ford’s dystopia, when the robber-barons are sitting atop their mountains of widgets that they’ve produced in their lights-out factory, don’t you think one of them might want to buy a sincere story? Wouldn’t they be interested in seeing a movie, or going to church on Sunday, or reading a book? When all of your basic needs are met, these higher-level needs will all see more demand. They’re also hard to automate. Some things have value because they’re done by people. Some things would be worth less if you did automate them:

  • Relationships (with real people)
  • Religion
  • Sports
  • The Arts
  • “Home Cooked” or “Hand-Made”
  • Stories (of origins, extremes, rescues, journeys, relationships, redemption, and the future)

Do you recognize that list? That’s the list of things we do when we’re finished with the drudgery of providing for our survival. We cheer for our sports team on a Sunday afternoon, or go and see an emotional movie on Friday night. Some people buy $200 axes (or iPhones, anyone?) because they come with a fascinating story that they can re-tell to their friends. (Bonus points if it’ll get you laid.)

Ford scoffs at the idea of a transition to a service based economy. He suggests implementing heavy taxes on industry and redistributing that to people who otherwise would have been doing the job the robots are doing, just so they can buy the stuff the robots are producing. He can’t see anything but an economy based on the consumption of material goods. I say: go ahead and automate away the drudgery of daily existence, make the necessities of life so cheap they’re practically free, and let’s get on with building real wealth: strong relationships, a sense of purpose, and a society that values life-long self improvement (instead of life-long accumulation of crap). By making the unimportant stuff less valuable, automation is what will free us to focus more on what’s important.