What Is a 3D Sensor?
A 3D sensor captures the spatial structure and surface of an object or scene. In addition to the two-dimensional location information in the x and y directions, it also determines the distance or depth in a third spatial direction (z). This creates three-dimensional data, for example in the form of a depth map or point cloud. On this basis, dimensions, positions, volumes and the surface geometry of objects can be determined.
When Is 3D Image Processing Used?
2D image processing captures features such as brightness, color and contour within a single plane. If additional information about the position, height or shape of an object is required, 3D sensors are used. The measurement data enables 3D object detection, volume determination, robot guidance and automated quality control. Typical applications include bin picking, depalletizing, robotic pick-and-place processes and 3D surface inspection.
2D Image Capture
3D Image Capture
Typical Applications with 3D Sensors
Bin Picking
In bin picking, a robot picks unordered objects from a bin or crate. A 3D sensor captures the position and orientation and generates a 3D point cloud of its visible surface. The robot system can then determine the suitable gripping position and remove the components in a targeted manner. This method is particularly suitable for objects with complex geometries or varying positions.
Depalletizing
3D sensors support the depalletization of boxes, containers or products from a pallet. Spatial data is used to detect height differences, object positions and stacking patterns, so that the objects can be picked up precisely by the robot. This allows even irregular or mixed loads to be processed reliably.
3D Surface Inspection
Dents, deformations, assembly errors and height differences can be accurately detected with 3D surface inspection regardless of color or contrast. The three-dimensional measurement data is therefore suitable for automated quality control as well as for checking dimensional accuracy and completeness.
Fundamentals of 3D Imaging
3D sensors provide the measurement data as a 3D point cloud and can also output intensity values depending on the sensor model. Industrial 3D image processing uses 2D/3D profile sensors and snapshot sensors, among others.
2D/3D profile sensors project a laser line onto the measured object and record individual 2D height profiles. The relative movement between the sensor and object combines the sequentially captured profiles into a complete 3D point cloud. This method is particularly suitable for continuous measurement of moving objects, for example on conveyor belts or in production lines.
3D sensors capture the entire measuring volume in a single recording. To create a complete 3D point cloud, the object must not move during capture. This method is therefore particularly suitable for high-resolution measurements of static objects as well as for capturing the finest surface details.
Coordinate Systems for Processing 3D Data
Coordinate systems are of central importance for the spatial processing of 3D data. The measuring points recorded by the 3D sensor refer to the sensor’s coordinate system. The sensor coordinate system defines the position of each measurement point in relation to the position and orientation of the sensor and forms the basis for further processing of the depth data.
For robots, plants or image processing systems to use the collected data, it must be transferred to a global or plant-related coordinate system. This allows object positions, movements and gripping points to be clearly determined within the application.
Hand-Eye Calibration for Robot Vision
For robot vision applications, the coordinate systems of the 3D sensor and the robot must be linked to each other. The hand-eye calibration determines the geometric transformation between the two systems. This allows the robot to interpret the 3D positions recorded by the sensor in its own coordinate system and move to them in a targeted manner.
Depending on the application, a distinction is made between two variants:
Eye-to-Hand
In the eye-to-hand procedure, the sensor is permanently mounted in the working area and observes the robot’s working range. The calibration determines its position and orientation relative to the robot base.
Eye-in-Hand
With eye-in-hand, on the other hand, the sensor is located directly on the robot arm and moves with it. Its position relative to the tool center point (TCP) is determined here.
Calibration takes place via defined reference objects or calibration patterns, which are recorded by both the robot system and the 3D sensor. The transformation between the coordinate systems is calculated from the captured positions. The detected image coordinates can then be transferred from the sensor coordinate system to robot coordinates. Even small deviations in the calibration can lead to position errors in the gripper control or path planning.
Hand-Eye Calibration with M3DS Series Sensors
Various project and calibration examples are available on GitHub for integration into robotics and image processing systems. These include reference implementations for hand-eye calibrations and transformations of 3D coordinate systems. This allows typical integration and calibration processes to be traced and the examples provided can be used directly for your own application.
Working Distance and Measuring Range of 3D Sensors
Working Distance
Measuring Range
The measuring range of a triangulation-based 3D sensor is shaped like a pyramid stump. As the distance to the sensor increases, the measuring field in the x- and y-direction increases. This allows larger objects or scenes to be captured in the far range.
At the same time, the measuring points are spread over a larger area, so that the distance between the neighboring points increases. At close range, the point density of a 3D point cloud is therefore higher and fine structures can be captured in more detail.
The depth resolution in the z-direction also depends on the working distance. As the distance increases, the geometric sensitivity of the triangulation decreases, so that small height differences in the near range can be detected better.
How Is a 3D Point Cloud Created?
3D sensors use structured light to project light patterns onto an object. The deformations of these patterns give conclusions about the contour of the object. A camera then captures the deformation of the pattern on the surface. From their displacement, the sensor uses optical triangulation to calculate the depth information and generates a 3D point cloud.
Step 1 – Projection of a light pattern
The built-in light engine projects sequentially defined light patterns onto the surface of an object. The pattern serves as a defined reference structure for the subsequent depth calculation. wenglor’s 3D sensors use blue light, as its shorter wavelength enables higher optical precision and at the same time is less sensitive to ambient light. This allows even fine surface structures to be captured in a stable and high-contrast manner.
Step 2 – Deformation of the Light Pattern on the Object
If the structured light pattern hits an object, it is distorted spatially depending on its geometry. Differences in height, edges and indentations cause the projected stripes in the camera image to shift. The deformation of the pattern contains information about the shape and depth of the object.
Smooth, textured or highly reflective surfaces can affect the quality of the pattern image and thus also affect the subsequent depth calculation.
Step 3 – Camera Captures the Light Pattern
The camera then captures the projected light patterns from a specified angle. During capture, the system analyzes the exact position and deformation of the projected patterns in the image. A complete data set of the scene is created from several recorded individual images. How clearly the patterns can be seen in the camera image depends, among other things, on the exposure time, surface reflection, ambient light and optical resolution of the sensor.
Step 4 – Calculate the Depth Value by Triangulation
The camera and the projector view the measurement object from different angles. Together with a surface point, they form a geometric triangle. Since the positions of the camera and projector are known to each other, the sensor can calculate the depth value of each individual measurement point from the shift of the recorded patterns.
This process is called optical triangulation and forms the basis for the spatial reconstruction of the visible object surface.
Step 5 – Creating the Point Cloud
Based on the camera calibration, the sensor generates a 3D point cloud from the calculated depth values. Each measuring point is described by its x-, y- and z-coordinates and thus has a unique position in the space. This point cloud thus represents the surface structure of the object true to detail.
Further information such as intensity or so-called confidence values can also be stored. While intensity data provides information about the brightness or reflectance properties, the confidence value indicates the quality or reliability of an individual measurement point.
Processing 3D Data
In structured light projection, different reconstruction methods can be used to generate 3D data. The stripe method and the phase method differ in the type of projected patterns and their evaluation. Both methods use gray code sequences, which allow the projected light patterns to be clearly assigned to the captured image points. This assignment forms the basis for calculating the depth values. Which method is suitable for a measurement task depends, among other things, on the required recording time, the condition of the object surface and reflections or ambient light.
Stripe and Phase Method Comparison
Stripe Method
With the stripe method, clearly separated and high-contrast light stripes of different widths are projected onto the object surface. Due to the high contrast, the projected patterns can be seen particularly well on dark or highly reflective surfaces. At the same time, the binary patterns allow very short exposure times, making the method particularly suitable for demanding surfaces.
Phase Method
The phase method projects sinusoidal light stripe patterns with different phase shifts onto the object surface. Instead of individual stripes, the continuous brightness distribution of the projected patterns is evaluated. This allows the position of a measurement point to be determined with sub-pixel accuracy, enabling particularly smooth, high-resolution and detailed 3D data.
Visualization of the 3D Point Cloud
False Color Representation
In the case of false color representation, different colors are assigned to the depth values of the 3D point cloud. This visualizes height differences and object contours, allowing quick interpretation of the measurement data.
Intensity Image
In addition to the depth values, the sensor can generate an intensity image. This corresponds to a classic gray value recording and contains information about the reflection properties of the surface.
Confidence Image
The confidence image shows the quality or reliability of individual measurement points within the point cloud. This allows areas with uncertain depth calculations or poor signal quality to be detected and evaluated in a targeted manner.
Recording Parameters of 3D Sensors
Exposure Time
Gain
Subsampling
Accuracy specifications according to VDI/VDE
Stack size
The stack size indicates how many individual images or pattern projections are used for a 3D image. A larger stack size provides more image information and can reduce measurement noise. At the same time, the recording time is extended, which reduces the maximum achievable capture speed.
Other 3D Technologies
Passive stereo
In this procedure, two cameras view the same object at an angle. By capturing the same features from both cameras, the spatial position of the feature in the room can be calculated from their image shift. The difficulty here is identifying the same point with both cameras. This method is suboptimal when viewing a low-contrast surface such as a white wall, for example.
Active stereo
The structure is the same as that of the passive stereo. In addition, a pattern (e.g. randomly distributed points) is projected onto the surface. This makes it easier to assign a point from both cameras.
Time-of-flight
In this procedure, the distance between the object and the sensor is determined based on the transit time. The sensor emits light pulses that hit an object and are reflected by it. The distance is determined depending on the duration of the reflection of the light pulses. This allows depth information such as structures or distances of objects to be determined.
Laser triangulation
In laser triangulation, a laser projects a line onto the object that is captured by a camera at a defined angle. The height of the object surface can be determined by the shift of the laser line in the camera image. As only one profile is captured at a time, either the object or the sensor must be moved to generate complete 3D data. The method is characterized by high accuracy, but is dependent on a defined relative movement during recording.
Comparison of Technologies
| Structured light | Passive stereo | Active stereo | Time of flight | |
|---|---|---|---|---|
| Resolution |  |
 |
|
 |
| Accuracy | |
|
|
|
| Ambient light | |
|
|
|
| Reading speed |  |
|
|
|
| Low-contrast objects | |
|
|
|
| Obstruction/shading | |
|
|
|
|
Resolution
|
|||
|---|---|---|---|
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Accuracy
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Ambient light
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Reading speed
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Low-contrast objects
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Obstruction/shading
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
Integration in Industrial Systems
For a 3D sensor to be used in industrial applications, it must be able to be seamlessly integrated into existing automation, robotics and image processing systems. In addition to the actual 3D data acquisition, standardized interfaces, software libraries and flexible integration options play a central role here. Modern 3D sensors offer extensive software and communication interfaces for this purpose, which enable easy integration into a wide range of system environments.
