Author Archives: Scott Whitlock

About Scott Whitlock

I'm Scott Whitlock, an "automation enthusiast". By day I'm a PLC and .NET programmer at ETBO Tool & Die Inc., a manufacturer.

Why good ladder logic looks like it was written by an 8 year old

When traditional PC programmers see ladder logic, they think ladder logic programmers are terrible programmers. Being both a .NET developer and a ladder logic programmer, this has caused me a lot of frustration and confusion over the years. I have one foot in each world, and yet I choose to write C# programs one way and ladder logic programs another. Why?

Let’s ignore the fact that most traditional programmers just don’t grok ladder logic at all, because their minds think about programs sequentially rather than in parallel. The real reason they hate ladder logic is because ladder logic programmers avoid things like loops, indexed addressing and subroutines. To them, this means you’re programming at the level of an 8 year old.

The thing is, I know how to use loops, arrays, and subroutines, not to mention object oriented programming and functional programming constructs like s-expressions, closures and delegates. Still, I choose to write simple and straightforward ladder logic. Why would I, an experienced programmer, choose to write programs like an 8 year old? Do I know something they don’t?

I spend a lot of time trying to get people to think about why they do things a certain way. Everyone wants that simple rule of thumb, but it’s far more valuable to understand the first principles so you can apply that rule intelligently. Decades of computer science has given us some amazing tools. Unfortunately, a carpenter with twice as many tools in her tool box is simply twice as likely to pick the wrong tool for the job if she doesn’t understand the problem the person who invented that tool was trying to solve.

The first time you show a new programmer a “for” loop, they think, “Amazing! Instead of typing the same line out a hundred times, I can just type 3 lines and the computer does the same thing! I can save so much typing!” They think this because they’re still an idiot. Don’t get me wrong, I was an idiot about this too, and I’m still an idiot about most things. What I do know, however, is that for loops solve a much more important problem than saving you keystrokes. For loops are one of many tools for following the Once and Only Once (OAOO) Principle of software development.

The OAOO principle focuses on removing duplication from software. This is one of the most fundamental principles of software development, to the point where it’s followed religiously. This principle is why PC programmers look at ladder logic and instantly feel disgust. Ladder logic is full of duplication. I mean, insanely full of duplication. So how can you blame them? God said, “let there not be duplication in software,” and ladder logic is full of duplication, thus ladder logic is the spawn of Satan.

That’s because programmers who believe the OAOO principle is about removing duplication are idiots too. Don’t they ever wonder, “why is it so important to remove duplication from our code?” Should we really worry about saving a few bytes or keystrokes? NO! We focus on:

  1. Making it do what it’s supposed to do
  2. Making it obvious to the reader what the program does
  3. Making it easy to make changes when the requirements change

… in that order.

In fact, #3 is the real kicker. First of all, satisfying #3 implies you must have satisfied #2, so ease of understanding is doubly important, and secondly, satisfying #3 implies you can predict what will change.

Imagine if you have to print the numbers from 1 to 5. If I asked a C# programmer to write this, they’d likely write something like this:

for(var i = 1; i <= 5; i++)
{
    Console.WriteLine("{0}", i);
}

… of course I could write this:

Console.WriteLine("1");
Console.WriteLine("2");
Console.WriteLine("3");
Console.WriteLine("4");
Console.WriteLine("5");

Why is the first way better? Is it because it uses fewer keystrokes? No. To answer this question, you need to know how the requirements of this piece of code might change in the future. The for loop is better because many things that might change are only expressed once. For instance:

  • The starting number (1)
  • The ending number (5)
  • What to repeat (write something to the screen)
  • What number to print
  • How to format the number it prints

If the requirements of any of these things change, it’s easy to change the software to meet the new requirements in the first case. If you wanted to change the code so it prints every number with one decimal place, the second way clearly requires 5 changes, where the first way only requires one change.

However, what if the requirements changed like this: print the numbers from 1 to 5, but for the number 2, spell out the number instead of printing the digit.

Okay, so here’s the first way:

for(var i = 1; i <= 5; i++)
{
    if(i == 2)
    {
        Console.WriteLine("two");
    }
    else
    {
        Console.WriteLine("{0}", i);
    }
}

… or if you wanted to be more concise (but not much more readable):

for(var i = 1; i <= 5; i++)
{
    Console.WriteLine(i == 2 ? "two" : i.ToString());
}

Here’s the change using the second way:

Console.WriteLine("1");
Console.WriteLine("two");
Console.WriteLine("3");
Console.WriteLine("4");
Console.WriteLine("5");

Here’s the thing… given the new requirements, the second way is actually more readable and more clearly highlights the “weirdness”. Does the code do what it’s supposed to do? Yes. Can you understand what it does? Yes. Would you be able to easily make changes to it in the future? Well, that depends what the changes are…

Now think about some real-life ladder logic examples. Let’s say you have a machine with some pumps… maybe a coolant pump and an oil pump. Your programmer mind immediately starts listing off the things that these pumps have in common… both have motor starters with an overload, and both likely have a pressure switch, and we might have filters with sensors to detect if the filters need changing, etc. Clearly we should just make a generic “pump” function block that can control both and use it twice, right?

NO!

Look, I admit that there might be some advantage to this approach during the design phase if you had a system with 25 identical coolant pumps and your purchasing guy says, “Hey, they don’t have the MCP-1250 model in stock so it’s going to be 8 weeks lead time, but they have the newer model 2100 in stock and he can give them to us for the same price.” Maybe it turns out the 2100 model has two extra sensors you have to monitor so having a common function block means it takes you… 20 minutes to make this change instead of an hour. We all know how much you hate repetitive typing and clicking.

On the other hand, when this system goes live, making an identical change to every single pump at exactly the same time is very rare. In fact, it’s so rare that it’s effectively never. And even if that were to ever actually happen, the amount of programming time it actually saves you is so tiny compared to the labor cost of actually physically modifying all those pumps that it’s effectively zero.

However, since these are physically different pumps, you’re very likely to have a problem with one pump. When your machine is down and you’re trying to troubleshoot that pump, do you want to be reading through some generic function block that’s got complicated conditional code in it for controlling all 50 different types of pumps you’ve ever used in your facility, or do you want to look at code that’s specific to that pump? And maybe the motor overload on that pump is acting up and you need to put a temporary bypass in to override that fault. Do you really want to modify a common function block that affects all the other pumps, or do you want to modify the logic that only deals with this one pump? What’s more likely to cause unintended consequences?

So this is why ladder logic written by experienced automation programmers looks like it was written by an 8 year old who just started learning Visual Basic .NET last week. Because it’s better and we actually know why.

How Automation is Shaping our Society

Try this on for size:

We’ve been automating for hundreds of years now. The industrial revolution caused a migration of workers from agriculture into the cities to work at factory jobs, and workers that are displaced by new technologies will find new work that didn’t even exist a few years ago.

I assume if you’re reading this article that you’re involved in automation in some way, so you’d actually want to believe that statement, and arguing against what someone wanted to believe would be pointless. I’m going to do it anyway. That statement is wrong. This time it really is different.

To explain this I need you to consider what motivates people here in the “west”. Basically we have some form of regulated capitalism. To boil that down, it means you can own things, and you are allowed to keep some fraction of the proceeds that are generated from those things. This actually applies to almost all of us, even if most people don’t think of it that way. It’s obvious to a farmer: you own land, buildings, and equipment, you grow things and sell them, and after tax you hope to end up with some kind of a profit. Ok, perhaps farming isn’t a great example because there are so many government subsidies involved, but the principle is the same with small business owners, and even employees.

Employees? Most employees don’t think of themselves as capitalists because they can’t see the capital they’re using to generate a profit, but it’s right there in the mirror. You are your own capital. Ever since abolition, this is capital that nobody could take away from you. It’s the first and primary social safety net. No matter how penniless you were, barring illness or infirmity, you had this basic nest egg of capital you could always draw from to bootstrap your life. Most people mistake capital for money, and that’s why they don’t see themselves as capital. After all, you can’t “spend yourself”, can you? Actually, going back hundreds of years, you could. Most slaves around the Mediterranean hundreds of years ago were slaves because they incurred debts that they couldn’t pay off, so they became the property of whoever they owed the debt to, until they could work off their debt.

If it helps, think of the human body as a machine that turns food into… various useful things more valuable than food. Farmers turn small amounts of food into larger amounts of food. Carpenters turn food and wood into houses or furniture. Quantum physicists turn food into transistors and lasers, and you, dear reader, perhaps you’re a machine that turns food into PLC programs. In turn, we trade these things for various useful things that other people have created. Capitalism.

Now I’m a fan of regulated capitalism because it’s an efficient way to organize lots of machines (us) into producing lots of valuable things like cars, houses and episodes of Game of Thrones.

Now here’s the weird part. There’s a huge incentive to use your capital to acquire more capital, which you can then use to acquire more capital, and so on, but very few people do this. You would think that someone who finished 13 years of schooling at the age of 18, worked 47 years and retired at the age of 65, making, let’s say, an average modest wage of $30,000 per year in present-day dollars would have had the foresight to save some of that $1.4 million for their retirement, but it’s clear that many don’t. In fact there are many people with an income far higher than that who not only don’t save any, but go into significant debt and either declare bankruptcy or become virtual slaves to credit card companies. It’s so incredibly common and has such a negative cost to society that governments actually force workers to save portions of their paycheque every week into a government pension program and then pay them a stipend when they retire. I’m not familiar with the way this works in the United States, but in Canada this is referred to as the Canada Pension Plan, and it’s supplemented by something called Old Age Security that kicks in a few years later. This is despite the fact that anyone who bothered to squirrel away 18% of their net paycheque for their entire career into a tax sheltered retirement savings plan and invested it in mutual funds would have a very comfortable retirement – much more comfortable than living on a government pension.

Now part of me thinks this is fine: you made your bed, now lie in it. But this affects everyone, even the wealthiest capitalists. The most basic of government services are the ones that wealthy people need most: military, police (criminal law) and the enforcement of contracts (civil law). These three services of government are what give people the ability to own things. The military protects it from external threats, the police protect it from people inside the country (thieves and vandals), and civil law settles disputes about who owns what.

We keep hearing that wealth inequality is a bad thing, but that can’t be absolutely true. If our system is working, it has to reward the people doing more valuable things with more money, so the only way there could be income equality is if everyone was doing something equally valuable, and we’re not. There should be a way for me to make more money by working harder, smarter, or differently than I am now. That’s the incentive to be more productive.

In fact, that’s what really matters: does the average person believe they can improve their standing? Because if they don’t, they get unruly and do wild and crazy things. Things that make wealthy people uneasy because in the west those unruly people can really mess with the government that’s providing all those military, police, and civil services they depend on.

Imagine you work in a factory in the Midwest U.S. that makes air conditioners. Chances are, you don’t think of yourself as a machine that turns food into air conditioners. You’re not thinking about how to make that machine more efficient, or more valuable. You’re already working 6 days a week, and your family never sees you. All you know is that sooner or later the guy who drives the fancy BMW is going to move your job to another country, or replace you with a robot, and since all the other plants around here have closed, you might not be able to send your kid to college. How would you feel? Maybe you’d be inclined to vote for a politician that promised to punish companies that moved their factories to Mexico.

I think the crux of the matter is that this worker no idea what to do. The incentives are still there: learn a new skill, invest in yourself, be more productive. But few people do it, for the same reason that few people save for their own retirement.

I’ve spent a few years around people who’ve been running small businesses, and I’ve tried to pay attention. It took me years to really understand that there was nothing magical about running a business. That’s because, like almost everyone else, I was brought up with the idea that innovative geniuses come up with brilliant new ideas and start companies that make billions of dollars. Outside of a few small cases, that’s simply not true. Look hard enough and you can find an industry that’s in demand and growing. If the demand is high, there will always be companies in that industry that are poorly run but still make a profit. You can make money simply by doing the same thing as everyone else and simply not being the worst at it. That’s how capitalism works – it gives you incentives to provide products and services that are in demand.

I have a relative that got laid off many years ago. There was a jobs program where they gave him classes on how to start a small business. He learned how to keep books, write an invoice, and how to do his taxes. They hooked him up with a small business loan. A few months later he’s running his own business and a couple years after that he’s hired an employee. Now he has the opportunity to invest in himself, like buying better equipment and improving his skills.

Let’s say you’re a PLC programmer. Your company likely pays you upwards of $50,000 a year. How much did they spend on your computer? Did they cheap out? Does it make any sense to handicap a $50,000 a year resource with a cheap laptop? If you were in business for yourself, you’d quickly realize there aren’t many things you could invest in that would make you a more efficient or valuable PLC programmer, but a faster computer is a no-brainer.

Automation is increasing productivity and with self-driving trucks and expert systems being developed, the rate of productivity increase is set to explode. However, these are expensive investments and there’s no way for displaced workers to take advantage of this automation. If I gave a truck driver a bigger truck, they produce more value per mile driven, but if I replace the driver with a computer, they have no value at all.

Increased productivity stopped producing higher wages back in the early 70’s. A bank teller makes the same now as they did back then (adjusted for inflation) even though most of the drudgery has been offloaded to ATMs. In fact, ATMs allowed banks to open more smaller branches and the demand for tellers to staff those branches has actually increased the number of tellers total, but despite automating the simple tasks and increasing demand for tellers, they’re not making any more in wages.

The same people who are currently blaming immigration and outsourcing for their problems are soon going to realize that automation is what’s really eating their lunch. Unlike in the industrial revolution where displaced workers could participate in this new economy by switching from farming to factory work, during this transition workers will either lose their jobs and have to completely re-skill, or at best they’ll keep their jobs but not see a penny more for their increased productivity.

That’s because old automation made people more valuable, but new automation seems to make them less valuable. That means it’s devaluing the one bit of capital they have.

This is where someone usually suggests a universal basic income so everyone can share in the increased productivity without everyone contributing to it. I’m not convinced the numbers add up. What we really need is to encourage this idea of viewing yourself as capital, not as an employee. An incentive and a safety net for people starting a small business should be less expensive and more effective than paying people to sit at home. How about teaching this stuff in school (I figure teachers are pretty clueless about starting a business). How about making it easier to start a business than going on social assistance? How about making in-demand skills training free?

I’m glad we’re talking about this because it does matter. A lot of this is tied in with what’s going on in the world right now. There’s a general sense that the next generation won’t be as well off as their parents’ generation, and that’s pretty much unprecedented. That promise that anyone could make something of themselves is slipping away, and we need that back.

Intellectual Property and Control System Integrators

I’m not a lawyer and what follows is not legal advice. What I’d like to do here is point out some minimum things that you need to be aware of if you’re doing control system integration when it comes to intellectual property. I hope you can appreciate that there are subtle details that can affect the reality of your situation and you should consult a lawyer depending on your situation, geographic location, and product.

If you do control system integration for a living then you deal every day with intellectual property. Ladder logic (or structured text, etc.) is code, and code is automatically copyrighted. Electrical designs are also intellectual property. The important thing to realize is that somebody owns that intellectual property, and people are sometimes surprised by who owns it.

Here’s a typical situation: you work for a control system integrator and they’re doing a fixed price (or “turn-key”) project for a customer. In this case the ownership of the copyright should be spelled out in the contract, and most control system integrators will specify that the copyright belongs to the integrator, but part of the purchase price of the project covers a license to use that software in their facility. There are very good reasons for this. The integrator works on many different projects for many different customers and over time you’re going to see the same problem over and over so you want the ability to maintain an internal library of tested code that you can re-use on future projects. Not only does this make you more efficient, but it’s almost impossible to avoid writing the same code again when you see a problem that you’ve solved before.

In this case the customer, to protect themselves, needs to insist on a license that will cover all of their needs, including the ability to see and modify the code, including having someone else come in and modify it, though only for use in that facility or on that machine, as the case may be.

Now consider a very different situation. You’re a control system integrator doing a time-and-material project for a customer. They have their own internal PLC ladder logic standard, including a library of function blocks, and they know how they want everything written. They give you a sample program and tell you to start with that and modify it to suit the machine you’re building. This is a completely different case. The copyright of the finished PLC program is almost certainly owned by the customer, not the integrator. The integrator and the customer should both be making sure that the contract specifies this in writing. This creates some interesting differences for the integrator.

The integrator cannot (and should not) be using their own internal library of code on this project, or the customer could end up claiming copyright on the integrator’s code. Even more importantly, the integrator should definitely NOT take code they see in the customer’s standard library and put it in their own internal library. Here’s what could happen: you take code from customer A and put it in your library. You use that code on a project for customer B, and it happens that customer B is a competitor to customer A. Customer A’s lawyers can now use the legal system to prevent customer B from using that machine without paying royalties to customer A.

You may be wondering, “how would customer A even find out that this happened?” There are several ways… for one, people in the control system industry move around. A lot. Trust me, it’s a small group of people and everybody knows each other. It’s not a big stretch for someone who worked at customer B to end up at customer A. For another, sometimes employees just talk. I’m writing this because just the other day I witnessed an employee from a local integrator say out loud that they’d done just this… took some function blocks from their customer and used it on other projects for other customers… and they said this in front of the customer they took them from! Now, the representative from the customer seemed to think this was OK, but I doubt their corporate lawyer would agree, and I doubt this agreement was in writing. If some future customer of the integrator ever got in trouble legally, what do you think would happen to that integrator?

I realize it’s sometimes hard to adjust from a university environment with the free flow of ideas, where everyone’s downloading copyrighted movies and songs from peer-to-peer networks all day long, and then work in a corporate environment where the stakes are much higher, where lawyers and executives on both sides would love to find some leverage over their competition. Seriously, it’s not worth your career. Take a moment to school yourself on intellectual property law and how it affects your profession. This can be a big deal. Be careful.

Don’t Pay Retail For…

There’s a certain class of product for which you’re much better off to buy online. Specifically, you’re looking for things that are commodities (but not dollar-store items), durable, small, and lightweight. Here are some things where you’ll find a much better deal on e-Bay, etc.:

  • Brother-compatible labeling Tape (aka TZ-Tape) as low as $5/cartridge
  • Ink Cartridges
  • Button cell batteries (leading brand CR2032’s for less than $1 ea.)
  • AAA Duracell batteries, packs of up to 100
  • Car lighter USB adapters (especially the high power 2.4 amp ones)
  • All electronic parts, obviously

For AA batteries, apparently there’s evidence that Costco’s Kirkland brand AA batteries are actually rebranded Duracell batteries, so I can’t find a better price than them online.

Since practically all electronics parts are made in China, you can get them on AliExpress.com for extremely low prices and you can usually get free shipping. I regularly purchase Arduino compatible or ESP8266 boards, prototyping supplies, power supplies and sensors mail order from AliExpress with free shipping, all for less than $10, sometimes as low as $1.80 with free shipping, and the only down-side is that you have to wait a month or two.

If you need it faster, and it’s a more popular item, it’s likely there’s a small importer that has some for sale on e-Bay in your own country, for a slight markup. This is true for batteries, chargers, ink and tape cartridges and usually for popular microcontroller boards.

TwinCAT 3 Tutorial Complete

Over the past year I’ve been working on a TwinCAT 3 Tutorial, and with the posting of the 13th chapter about the TwinCAT 3 Scope View, I’m considering it “complete.” I’ve covered all the major topics that I wanted to cover.

Writing a tutorial like this was actually a learning experience for myself. Before I made any statement about the software I tried to validate my assumptions first, and many times I learned that I had preconceived notions about how it worked which may not have been completely accurate. If you’re a TwinCAT 3 programmer, then you need to know your tool set inside and out. If you’re not sure how something works, it’s best to setup a quick test and try it out. That’s one of the advantages of PC-based software: if you want to run a test, your office PC is a laboratory environment. Write some code and run it.

Thank you to everyone who offered feedback on the existing sections, and to those of you who kept pushing me to continue writing more. I sincerely hope this can be that missing link to take people from the world of traditional PLCs into the more interesting and ultimately more powerful world of PC-based control.

It looks like TwinCAT 3 has a viable future ahead of it. Interest seems to be growing over time (according to Google Trends, anyway):

That makes TwinCAT 3 a valuable skill to learn if you’re an integrator, and an interesting technology to investigate for existing plants.

How to Write a Big PLC Program

Staring down the barrel of a big automation programming project is intimidating. It’s hard to even know where to start. Even when you’ve done a few before, you’re only marginally more confident the next time.

I have quite a few big automation programming projects under my belt, so I think I can generalize the process a bit. Here goes:

1. Get the Prints

There’s almost no point in starting to program unless you have an almost final set of electrical drawings. If you don’t have them yet, push for them, and go do something else productive until you get them.

2. Create a Functional Specification

You don’t always have to write out a functional specification, but it at least needs to exist very clearly in your head. If at any point, you don’t know exactly how the machine is supposed to work in excruciating detail, stop writing code and go figure it out. Ask stakeholders, talk to operators, whatever it takes. Functional specifications are best written as a list of “user stories”. If you’re not sure what a functional spec should look like, check out Painless Functional Specifications by Joel Spolsky.

3. Shamelessly Copy

Identify what other projects you can find with logic that you can steal. Any code that works in another machine has the advantage of already being debugged. Don’t re-invent the wheel. (At the same time, never blindly copy logic without understanding it. Copying the code by re-typing it one rung at a time is still faster than writing it from scratch, and it’s a form of software review.)

4. Structure Your Project

Now you break open the ladder logic programming software and start creating your project. Pick your CPU type, setup the I/O cards based on the electrical drawings. Map your inputs. Plan out your program by creating programs or routines for each functional unit of the machine. Setup your fault summary rungs and your alarm logic.

5. Write the Manual Mode Logic

PLC logic is typically written “bottom up.” Manual mode logic is the lowest level of logic because it deals directly with individual functions in the machine. Advance cylinder. Retract cylinder. Home axis. Jog axis. While you’re writing the manual mode, this is when you take extreme care making sure that actions are interlocked so the machine can’t crash. If you’re using the Five Rung Pattern, this means paying attention to what goes in the Safe rung. Does cylinder A always have to advance before cylinder B can advance? The Safe rungs should reflect that. Make sure that even in manual mode, the operator (or you) can’t break the machine. Make sure to hook your faults and alarms into the applicable fault summary rungs and alarm logic.

6. Write Part Tracking Logic

Now that manual mode is complete, write the logic that tracks parts (and their state) through the machine. Remember, you should be able to run the machine in manual mode, and the part tracking should (ideally) work correctly. I know this isn’t always the case but surprisingly part tracking in manual mode can work 95% of the time. That means part tracking works based on the state of the machine. Closing the gripper with the robot in the pick position and the part present in fixture sensor on should latch a bit “remembering” that the gripper has a part in it.

Once you’ve written your part tracking logic, go back and use the part tracking and state bits to condition your Safe rungs. Don’t let the operator (or you) mistakenly open the gripper if the gripper has a part and isn’t in a safe position to let go of the part. Of course, you may need to add a way to manually override this (that’s what output forcing was created for), but in most cases you want to prevent improper operation.

Part of writing the part tracking logic is adding “ghost buster” screens. Operators often need to remove parts from a cell, and if the machine can’t detect their removal, then you have to provide the operator with a way to clear these “ghosts.”

At this point you’re actually ready to dump the program in and start testing out the machine electrically and mechanically. While it’s ideal to have a fairly complete program when you go onsite, we all know that’s not always entirely possible. At the very least you want to get to this point before startup begins.

7. Write the Auto Mode Logic

The complexity of your auto mode logic depends on what type of machine you’re programming. You’ll always need a cycle start and a cycle stop feature. Even if you’re in auto mode, you usually don’t want the machine to start until the operator specifically tells it to start. Once it’s running, we call this “in auto cycle.”

In simple machines, you can write the auto logic by filling in the Trigger rungs in your Five Rung logic. Start by putting the In Cycle contact at the beginning of the rung, and then writing logic after that which expresses when the action should take place. For instance, an advance reject part cylinder’s Trigger rung could be as simple as In Cycle, Part Present, and Part Rejected. As long as the Part Present tracking bit gets cleared once the cylinder is in the advanced position, then this is all the auto mode logic you need for this motion. Have the retract Trigger rung be In Cycle, No Part Present and Not Retracted.

More complicated machines need more complicated auto mode logic. If your machine has to perform a series of steps (even if some of them are in parallel) then consider using the Step Pattern. If your machine needs to choose between several possible courses of action (commonly seen in a storage and retrieval system) then consider using the Mission Pattern.

8. Review

It’s hard to write correct logic. Review your functional specification, point by point, and make sure your logic meets all of the requirements. Check your logic for errors. A fresh look often uncovers incorrect assumptions, typos, and outright mistakes. The earlier you find and fix problems, the easier they are to fix.

Make a list of everything you have to do during startup. Starting up a machine is time consuming and therefore expensive. Anything you can do to prepare saves you time and money.

Good luck, and keep your fingers out of the pinch points!


Price of Oil Affecting Alberta and Ontario

The recent drop in the world price of oil has caused a similar drop in the value of the Canadian dollar. Much of that has to do with the huge oil production in Alberta, so that province is being hit particularly hard right now. The “oil sands” in Alberta are particularly expensive to exploit, as oil sources go, so companies there are quick to make cuts when the price drops below their break-even point.

The previous run-up in oil prices saw a lot of migration out of manufacturing-heavy Ontario westward to Alberta, and control system specialists were a big part of that. It’s difficult to see them facing these challenges, and none of us want to be in that position.

Back here in Ontario, the pendulum looks like it’s swinging the other way. When rising oil prices drive up the Canadian dollar, it makes it a lot harder for Canadian manufacturers to compete in the export market, because their products are automatically more expensive. The recent drop in the Canadian dollar, if it persists, is going to mean growth in the manufacturing sector, especially along the 401 corridor in Ontario. Personally I’m looking forward to seeing some growth in what has been a relatively stagnant market over the past 10+ years.

Automation is poised to be a big part of the new growth in Ontario. At first companies are likely to be cautious about adding new capacity, likely adding temporary labour, but if the trend holds, I think we’re going to see lots of interest in automation projects throughout the area. Companies are still a bit leery of the economy, so they’ll be looking for flexible automation that can adapt to changes in demand and re-tool quickly. Technologies that can support flexibility are going to be winners.

While disappointing for Alberta, I’m excited to see what the future holds for Ontario.

If you do happen to be out in Alberta and you have discrete automation programming experience, and you’re interested in moving back to Ontario, make sure you drop me a line. We’ve experienced lots of growth ourselves in recent years, so we’re interested in hiring an experienced automation programmer for our Electrical Engineer position. We use TwinCAT 3 and C# and mentoring is available to get you up to speed with these technologies. Even if you’re local and you’re tired of the endless travel or 2 am service calls, then maybe this family-friendly company is right for you.

Start your own Automation Blog!

One thing I’ve discovered about automation blogging is that it’s a pretty lonely place. Don’t get me wrong, there are a couple gems out there, but I don’t find many people writing about what it’s like in the trenches wading through rungs of ladder logic. In the .NET world there are tons of programming blogs with posts about every issue you could ever come across. Why such a dearth of information in the automation space?

It occurs to me that blogging seems difficult to a lot of people, as if you need to be a web programmer. That’s totally untrue (I’m absolutely not a web programmer). In fact there are a ton of inexpensive and simple options out there.

Maybe people wonder what to write about. That’s easy. At first I worried about writing something people didn’t already know, but it turns out there’s no shortage of shiny new graduates looking for any toe-hold they can get in this industry. Try to think back to when you didn’t know anything. How did you figure out how to get online with a PLC the first time? Did you have to call the office to ask someone, and feel foolish because you didn’t know what a Null Modem cable was? We were used to asking our peers for help, but the new generation grew up looking up how to do things on the web.

Maybe you’re worried about the cost. Shared hosting plans are very inexpensive. I use DreamHost, specifically because they’re inexpensive (starts at $9 per month including domain name, unlimited storage and bandwidth), the hosting is rock solid, they offer one-click installs of blogging software (such as WordPress), and their technical support is excellent.

It would really be great if I didn’t feel like the only voice in this cloud. Grab a blogging account and chime in. Write about a hard problem and how you solved it. Disagree with me! Help me learn something new!

Offline Changes to a PLC Program

As a PLC programmer, you’ll often be asked to do a change to an existing system. If there’s a significant amount of functionality to be added, you generally get your changes ready “offline” and then do all the changes during a short window of time to minimize disruption to the production schedule.

If you’re using an Allen-Bradley PLC, the procedure is typically this:

  1. Get a copy of the latest program from the PLC (a.k.a. an “upload”)
  2. Make your changes to the offline copy, and write down every change you had to make
  3. Go online with the PLC and apply your changes as online changes

Step 3 is much safer than just taking your modified program and doing a “download”. That’s mainly because when you download, you’re not just downloading the program, but the memory state of the PLC as well. The PLC typically has to track things in memory (like recipe data, part tracking, data collection, sequence numbers, machine counters, etc.). If you do a download, you’re going to overwrite all those values with previous values, and that can cause a lot of problems. The other thing step 3 saves you from is simultaneous changes that were done online while you were busy making offline changes.

The only other option you have is upload-change-download, but you really have to shut the machine down for the duration to make sure that the internal state doesn’t change.

When I did a lot of Allen-Bradley programming, I didn’t question that. It’s just how it was. I remember visiting a plant one time for a service call, and the local maintenance person was a bit suspicious of what I was going to do (after all, I was a young kid who had never seen this machine before). He decided to quiz me a bit, and one of the things he asked was, “when you go online, do you download or upload?” I said “it depends,” but his answer was, “you never download.” I agreed that someone in a maintenance role should never need to do a download unless they’re replacing a CPU, or recovering from a corrupted PLC program.

Now that I mostly do Beckhoff TwinCAT 3 programming, I realized one of the benefits are that offline changes are a breeze. It’s due to the fact that TwinCAT 3 completely separates the program from the memory data. The program is stored in local files on your hard drive and compiled into a TMC file. The persistent data is stored in a different place on your hard drive.

When I want to do offline changes to a TwinCAT 3 project, here’s the procedure:

  1. Get a copy of the latest program
  2. Make your changes to the offline copy
  3. Copy changes back to the machine (keeping a backup, of course), rebuild, and activate configuration

This makes offline changes go a lot more smoothly, of course. I don’t have to copy and paste my changes in while online, so it takes less time and eliminates the possibility of a copy/paste error.

Since we also use Mercurial for version control, getting a copy of the latest program is a matter of pulling the latest from the source control, and copying it to the machine is a matter of pulling the offline changes to the machine. Any changes that were done in parallel can be merged with Mercurial’s built-in diff and merge utilities. (Note: I/O changes can’t be merged nicely, so if someone changed the I/O while you were doing your offline changes, you have to copy those changes in manually, but that’s rare and at least it tells you that it can’t merge them.)

This got me thinking that Allen-Bradley probably has a better way of doing offline changes that most of us just don’t know about. I know that you can do an upload without uploading the memory. However, it seems like it requires you to download both the program and data at the same time. I wonder if anyone out there knows how to do better offline changes to a ControlLogix. If so, I would be interested to know that.

The Dirty Secret About Housework

If you don’t have kids, I think I have an insight that you may find interesting.

It’s funny, but before we had kids, it’s amazing how much less housework got done. I remember the dishwasher didn’t get unloaded in a timely manner, or the floors didn’t get vacuumed as often as they should. The laundry didn’t get folded. The weird thing is, now that we have three kids, we’re way more on top of that stuff. Yet we have a lot less time. How is that possible?

There are a lot of things in our lives that distract us. The big time suckers have screens: smart phones are the new offenders, but before that it was the internet, video games, and of course television. Personally I’ve gone to excessive lengths to curtail these distractions, particularly at work. I leave my phone in airplane mode about 95% of the time. Email notifications are always turned off. I use a browser add-in that blocks my favorite websites except at specific times of the day, or blocks them after a specific number of minutes used each day. Overall, these small barriers to distraction really do make a difference, but there’s actually another much more potent barrier to screen distractions: young children!

Whether it’s one child or three, they demand attention almost constantly. You can’t get lost for 10 minutes on your phone if your two-year-old interrupts you every 90 seconds. Parents never “surf” the web, they go to a website, lookup whatever it is they need, and slam the lid on the laptop before the sticky fingers come around. God forbid you get the idea to play a video game… you’ll never get past the loading screen before you have to attend to some crisis.

When kids are around, you can’t do anything that requires concentration and focus (a.k.a. flow) because the constant interruptions prevent you from making progress. On the other hand, if you focus 100% on your kids, you’ll drive yourself insane. They’re always doing something that makes you slightly nervous, like playing too close to a lamp, crawling over the back of the sofa, or jumping on their bed (no matter how many times you’ve warned them). In fact, so called “helicopter parents” are just the parents that spend all their time focused on what their kids are doing. The rest of us realize that we need to ignore most of it, and just keep one ear open for the real dangerous stuff.

So as parents we’re left with having to do something that doesn’t require our full attention. Do you know what kinds of things are ideal? Housework. Laundry. Tidying. Cleaning. Taking out the garbage. These are the kinds of tasks that can be done with half your brain. It doesn’t matter if you get interrupted while you’re folding laundry, because you can pick it back up almost instantly. It takes just enough of our attention to keep us from actively focusing on the kids, but your subconscious still alerts you when they get (a) too noisy, or (b) too quiet. Best of all, it gives you a much needed feeling of productivity. Yes, housework is actually a coping mechanism for parents to keep their sanity.

So if you don’t have kids, and you wonder why you’re not the productivity powerhouse that you remember your Mom being when you grew up, don’t worry… she probably wasn’t that on top of things until she had you!