Search:     Advanced search


Discuss your PLC and automation problems for Allen Bradley, GE, Omron, Mitsubishi, Modicon, Panel Views, RS View, ABB, Adept Robots, Cognex etc. You can also submit and download sample PLC code.

If our forums and knowledge base still hasn't got your PLC operational, we recommend contacting PDF Supply and Electric for PLC repair and replacement parts.

Application - Vision Guided IAI IX Scara Robot with Cognex

Article ID: 72
Last updated: 13 Oct, 2010

Vision guided robots have been around now in the trade shows for quite some time and easily impress me to watch. Learning to do a vision-guided application myself was a bit scary I must admit, but nonetheless, I was very excited to get an application to use two products that I am very familiar with. I've used Intelligent Actuators IX series robots many times and really like what the robot can do. I've also used Cognex vision systems, and very familiar with vision systems and lighting. So why not make the two work together? That's what I've done finally! Granted, I have seen other applications such as Fanuc's Vision software, which is easy to use, but this application couldn't justify the cost of a six-axis Fanuc, instead we used a four-axis SCARA robot, which fits our application.


If you've never done a vision-guided robot, your probably asking yourself the same questions I asked when I got started. How do I get the vision system X, Y, and theta information in the form of pixels to translate into X, Y, and theta the robot can understand? I was really concerned that I would have to come up with my own complicated trigonometry formulas to be able to do this application. Luckily for you and me, Cognex saves the day with a trigonometry function built into the camera software called TRANSPATTERNTOWORLD in conjunction with a CALIB "Calibrate" function, more on this later.

The other thing I kept asking myself is how do I get the information from the Cognex to the IAI robot? With a little legwork, I found out that the IAI IX controller supports ASCII using serial by purchasing a dedicated RS-232 card with my controller. Cost was $550 for two RS-232 DB-9 ports installed in the IX controller. I don't advise that you use the DB-25 teach port, but rather fork the bill for the additional RS-232 ports so you can still have a DB-25 teach port for your laptop. I thought about using Ethernet, but I never got a warm and fuzzy feeling about getting Ethernet to work between the IAI robot and Cognex camera. So I went with what I knew, plain old RS-232 and ASCII. 


After establishing a direction, I waited until everything was assembled on the machine and went to work. The application I had was to find the theta of a part using the camera and reporting that information back to the robot so it could place the part in the correction orientation on to a dial index table. The part was bowl feed to the robot so the orientation was unknown. The robot would always pick up the part the same, then read the camera data, upon placing the part, a calculation was done to subtract the pickup theta from the theta the camera said the part was located. This information was used to determine the final theta correction factor needed to place and orient the part on the dial table. Originally, I misunderstood our Mechanical Engineer thinking that I would need to know the X, Y and theta of the part so I wrote all my software to handle all three positions even though I ended up only using theta (degrees). The camera was mounted upside down to the robot, this kind of concerned me at first and didn't know really how to handle this, but after using the calibration function in the Cognex camera, it seemed to not make any difference.


I started to develop my Cognex application first. I started with the robot calibration wizard program that Cognex has in their online support download section. After playing with this wizard, I end up only using the four-point calibration method from the wizard. The way this works is you teach four points with a pattern match tool of your part. I can't show you the part in this article for disclosure reasons, but picture a part with a pattern tool selected around it. For example, what I did was pick up the part in the robot gripper, ran the pattern match tool to get the X, Y, and theta in pixels of the part. While the robot was still holding the part, I looked at the real world robot position and typed them into the Cognex camera. So in my example, point "0" was 370.527, 323.711 pixels, which was equal to the robot X and Y value of 176.427, 69.727.



The calibration function needs four or more points entered to work correctly. You'll notice that theta is not included in the calibration. The theta is dealt with a little differently all on it's own. It's important that you move the part to the extreme portions of your field of view to get a better, more accurate calibration. Since vision lens have a natural "fish-eye" effect, teaching points in the center and towards the outside of the lens field of view will give you the best accuracy. Of course, you can use more than four points if you want, but four points was enough to give me an accuracy of about +- 0.1 mm. Because of where I had to mount my camera, I was forced into using an 8mm lens, which is a very bad lens to give off fish-eye distortion. If you can use a 25mm or a tele-centric lens (50mm), I would definitely recommend it if you can fit your camera mounting location using those size lenses. Use a better lens especially if you need better than +- 0.1 mm accuracy for your application.

You can see the four points that are calibrated in this screen shot. Since the robot can only move to these maximum pick up positions, I calibrated to the outside of the tool, not the outside of the screen.




Once your calibrated, next you'll need to throw the TRANSPATTERNTOWORLD function on your sheet. Basically it needs to be pointed to the calibration points you set earlier. Once is has these calibration points, the idea is you can feed this function any pixel pattern X, Y and it will tell you what the real world robot X, Y is of the part you just took an image of. I ended up separating my X, Y from my theta in my application because in the beginning of the project I was confused at what I was suppose to be doing at first. I found this more flexible in the end, so I created two different TRANSPATTERNTOWORLD functions but used the same calibration points. Mainly I needed this for a part lookup function I was using. My application needed to run different parts. So in the end it worked out ok, plus I was able to prove to myself I can do all three positions. (X , Y and Theta).



The pattern find tool in the Cognex camera reports the theta of the pattern. This is the same theta I needed to determine the rotation in degrees for my robot. It was accurate enough for me to use this value as the pattern tool found the rotation of the part. One thing I did notice was the pattern find tool does not report back a full 360 degrees. At degree 0 you will get a 0 Angle from the pattern find tool. At degree 179 you'll get a +179 Angle. At degree 182 you'll get a 178 Angle.


I didn't really find any fault with this logic, other than the robot might make a few extra turns to get 178 degrees. Or so it would make two complete revolutions sometimes if it found a part pattern that was at 178 degrees (angle), which was really 182 degrees to the robot, but I went ahead and sent the 178 degrees and made the robot do the two complete revolutions. I had enough cycle time in my application to allow this. At the end of my project I wanted to come back and write some logic to automatically correct this 178 degrees issue I had found, but I just ran out of time and it was working ok. So I never went back and wrote something that would fix 178 Angle to read 182 degrees.

With the vision application taking images, now solving X, Y and theta, and reporting pixel patterns to the TRANSPATTERNTOWORLD function, it's time to generate an ASCII strings to send to the robot. When working with ASCII, I found it easier to make the ASCII data separated or delimited as I send it to whatever RS-232 device I am dealing with. In this case, I borrowed the example from the Cognex robot calibration wizard, and used a TAB space delimiter. I also read a little bit about the WRITESTRING function in the Cognex online help. I found out that the data needed to always change in order for the function to generate the string and send it out. So what Cognex did in the robot calibration wizard was make a counter that would always change the string data on an image update. I thought this was a good idea, and I used the same idea. My first data character in my string is the counter value that is sent to the robot. I really just ignore this value once it arrives at the robot but it's a good idea for keeping the ASCII strings up-to-date. Next, I arranged my data in some orderly fashion. Once again modeling what I seen done in the Cognex robot calibration wizard software, I put the X, Y and theta information in cells. I also created a cell with a value "Stringf("%c",09)" This denotes a TAB SPACE in ASCII. Now I need to Concatenate all the string values. Basically if you look at cell E70 in my Cognex application you'll notice this function "Concatenate($D$64,$D$68,$D$65,$D$68,$D$66,$D$68,$D$67)" What I am doing here is compacting all the information I need into one big string to send to the robot. First I grab the counter value, then add a TAB SPACE delimiter. For X, I get the X value, and then add another TAB SPACE delimiter. I do the same for Y, by grabbing the Y value, and then another TAB SPACE delimiter. Finally I get the theta value. Once all of these values are concatenated, I can send this string to the robot, and let the robot dissect the data I need at the robot end.



Using serial from the Cognex to the IAI robot I was concerned how to trigger the camera. I wasn't sure at first if I would trigger the camera manually, or try to trigger the camera through RS-232 etc. So I thought about it a second, and went with letting the camera trigger continuously. I setup the Cognex camera in continuous mode within the Cognex Insight software. The vision system continuously sends data to the robot whether the robot is ready for it or not. Sounds risky doesn't it? Well it worked out pretty nice, I'll explain later. I even had some spare time during the project and wrote a loop program in the robot to always move to a position point that was accepting vision data. Do you know what that created? A robot that would follow my pattern find tool in REAL TIME! It was awesome to watch the robot arm follow the pattern I taught in the vision system. As I moved the part, the robot would follow my part as the vision system was triggered continuously to update the X, Y and theta information about the part.

Although triggering continuously sounds kewl to keep blasting data into outer space, there is one obvious problem I encountered. When I wanted to read the data from the camera, I had no idea where the robot actually started reading the string. Did it start reading at the beginning, in the middle? I handled the rest of the communication in the robot program, more to come later. Of course the second downfall I noticed was I had another camera on the network, and while browsing the second camera online, update times on the network where kind of laggy, but I expected to see some.

Before you can write a program in the robot side, you should probably get all your hardware working first. There are settings in the Cognex camera that will need changed. First of all you'll need an Expansion I/O unit for whatever camera you are using. The expansion I/O has the RS-232 connection you will need to connect to the robot.




In the Cognex software you'll need to setup your expansion port settings. I used Native mode and setup a basic serial communication settings that I am comfortable with.



You'll also want to change any end character settings while you are here I used "LF" or line feed as the end character. Click on Details under the Mode selector to set your end character.


Go ahead and click ok, you should be done with the Cognex camera at this point.

Next, you'll need to verify settings in the robot IX controller. One of things that bug me about the IX is the parameters. It's like a black art to understand what all IAI has going on in this controller. Luckily there is a nice appendix page in the Second Edition manual for the IX controller to look at. If you bought your serial card direct from IAI, they will already have the parameters set in the controller for the serial card to function. I ended up changing my baud rate, so I had to reference the IAI IX manual second edition to change my baud rate. I change my settings to: "28100013h" in parameter I/O 100 which means 38.4Kbs, data length 8, stop bit 1, no parity, com mode RS-232, enable receive after send, d-sub upper, 3 slot. All of that is explained on page 274 of the IAI IX manual.




Cabling between the Cognex expansion I/O and robot RS-232 serial communication card was a straight through cable, non-crossed, straight pin-to-pin connection. I used the adapter connectors IAI had supplied with the communication card, which adapted the communication card DB-25 to DB-9 connection. Now that you've got a basic understanding of the hardware and software settings, let's look at the robot program.


I wrote a sub routine in the robot to read from channel 2 and dissect the ASCII data and store them into string registers in the robot side, then finally converting them to something usable like a decimal value to be used in the point table of the robot. In the first part of the subroutine BGSR GETPOS, I set the end character I am looking for from the robot using the SCHA 10 command. In this case, character "10" which means LF or Line Feed. I set the line feed character above in the Cognex software. TMRD is a command in the robot that sets the timeout for the serial data. If no serial data is received when the channel has been opened for 30 seconds, then there will be a communication fault on the robot side. Now I am ready to READ channel 2. I do TWO READ commands in the robot program, the first READ command really means nothing other than to start reading the string. The first READ command could start reading the data at the beginning, or it might start reading the data in the middle of the string, I don't know for sure. What I do know for sure is this command terminates when the end string is received, in this case once the Cognex camera sends an LF, the READ command stops executing in the robot side, and goes to the next step in the robot code. If there is a communication error, the robot sets some bits in the temporary register "99". In the next part of the robot program, I monitor the temporary register for any flags that the READ command placed in there if there was a communication problem. If there was a problem, I turn on a fault flag "711" and bail out of the program. If not, I make sure the temp register is cleared, and go immediately for the second READ command. The second time I READ the data, will be at the beginning of the string. I know this because two steps above the first READ command terminated at the end character. The robot CPU scan time is much faster that 38.4 Kbs, so ending the first read command and starting a second READ command was not a problem for the robot. I haven't missed any data, nor I have noticed any of my robot data getting chopped because I was late reading the string the second time. Again, I run the same fault check "if equal" to monitor any temporary faults that were placed in variable 99 from the READ command. Now I am ready to close the port with a CLOS 2 command.




After we have read the string we now need to separate the string back out in a usable form the robot can understand. Using the SCPY command you'll have to start counting characters to know where the data is you want. At character 3 the X data starts which will be read as seven digit characters and three decimal characters. The seven and three come from setting the SLEN command. "7.3" So at character 3 stating reading. Don't forget a tab space characters. SCPY will get seven integer digits and three decimal digits. Continue to do this for all your data. The other part of the commands tells the robot where you want to store your parse, SCPY 50 3, means store the parsed data into column 50, which is a string register. By clicking on the green "S" in the IAI programming software while connected to the robot, you can monitor what strings are stored in the string registers. Organize your strings that you have extracted in a way you can remember the location easily. Once you have extracted all your data, it's now time to convert the string values to decimal format. Using a VAL command does this for you in the robot program. Store your decimal values into a real register of the robot. You don't want to lose any of the decimal information. The robot will need this for accuracy. I stored my X into real register 100, my Y into real register 101, and theta into real register 102. Now that we have the numbers converted, it's just a matter of copying them into a position register. Sounds pretty easy right? I thought so...



In summary, what did I learn? Vision guided robots using Cognex is pretty easy in conjunction with RS-232 and native ASCII. All of the pixel to real world calculations are done in the vision system, not the robot. So the information the robot receives is the actual X, Y position data it needs to move to that point. Remember that when you do teach a pattern in the vision system to your part, that this becomes the pickup point. In other words circle parts are really easy because the gripper pickup point is the same X, Y location as the shaft spline on your robot. If your part pickup point is offset from your robot spline X, Y you have two options to correct this. One solution would be to create a tool offset in the robot side to offset the end effector tool. The other option would be to teach your pattern to the center pick point on the vision side.


I would recommend RS-232 again for this application, I had no problems at 38.4 Kbs, nor did I have any problems putting the camera in continuous mode, and doing a double read in the robot program. I was happy with the results of the Cognex as well as the IAI IX robot. Don't forget to order a dedicated communication card for your robot, as well as the expansion module I/O that will come with a RS-232 port for your Cognex camera. Remember to set up the communication settings in your camera under Device > Serial Port Settings > Port 0. Set your ASCII mode to Native, ending character and communication parameters such as baud and parity. Make sure to double-check the factory parameters on the factory installed communication card from IAI. Set your communication parameters to match your communication settings in your Cognex camera using page 274 in the second edition IX controller manual as a reference. In my case, my parameter was "28100013" for I/O parameter 100.

In the future, I would like to try an Ethernet setup. I've been told this can be done. IAI has written a DVT to IAI IX controller tutorial, so I assume if DVT can talk native ASCII over Ethernet to the IAI controller, Cognex probably can also. You can download the DVT tutorial here. With Cognex now owning DVT, maybe I can pick on DVT's tech support if I ever wanna try this again using Ethernet. Looking at the DVT tutorial produced by IAI tech support, it appears that DVT has a "driver" already made for the IAI robot for vision guided, but understanding how Cognex functions work, I don't see a need to make a special "driver" for Cognex.


 Learning curve rating: 


Curve 0 = Walk in the park.

Curve 10 = Get out the scholastic cap and crash in the classroom. 



I think you need to know a great deal of knowledge with Cognex cameras, IAI IX programming, and how to manipulate ASCII strings. Once you have experience in all these areas, I think this type of application will become easier for you. Manipulation of ASCII strings comes in handy for a great deal of applications, not just vision guided robots. For example, I would use ASCII RS-232 to convert a bar code string into an integer in a PLC the same way I converted an ASCII string from the Cognex camera to a real number for the robot. The concept is the same.


Download the source code from this article here.

Article ID: 72
Last updated: 13 Oct, 2010
Revision: 1
Views: 13278
Comments: 2
This article was:   Helpful | Not helpful
Prev   Next
Applications      First Experiences

LadderLogic | 22 May, 2005 12:00 AM
Can IAI controller talk Modbus TCP/IP? Cognex cameras support this protocol by default, without any additional programming.
Posted: 12 years 9 months ago   | Permalink
Gabe | 25 Sep, 2008 12:00 AM
When I began my Masters in Controls my advising prof had a dream of a small fleet of rover like robots to map out campus. I spent a lot of time finding data on stereo vision cameras. Back in 2001, when I was looking into this stuff, there was a lot of discussion of the fish-eye lens effect and others that would limit the accuracy of any vision feedback sensor.

From this article I would assume that commercial camera systems have overcome this issue - to an extent. Other than the calibration procedure, did you need to do anything else?

Posted: 9 years 5 months ago   | Permalink

Prev   Next
Applications      First Experiences