What Is a 3D Sensor?
A 3D sensor captures the spatial structure and surface of an object or scene. In addition to the two-dimensional location information in the x and y directions, it also determines the distance or depth in a third spatial direction (z). This creates three-dimensional data, for example in the form of a depth map or point cloud. On this basis, dimensions, positions, volumes and the surface geometry of objects can be determined.
When Is 3D Image Processing Used?
2D image processing captures features such as brightness, color and contour within a single plane. If additional information about the position, height or shape of an object is required, 3D sensors are used. The measurement data enables 3D object detection, volume determination, robot guidance and automated quality control. Typical applications include bin picking, depalletizing, robotic pick-and-place processes and 3D surface inspection.
2D Image Capture
3D Image Capture
Typical Applications with 3D Sensors
Bin Picking
In bin picking, a robot picks unordered objects from a bin or crate. A 3D sensor captures the position and orientation and generates a 3D point cloud of its visible surface. The robot system can then determine the suitable gripping position and remove the components in a targeted manner. This method is particularly suitable for objects with complex geometries or varying positions.
Depalletizing
3D sensors support the depalletization of boxes, containers or products from a pallet. Spatial data is used to detect height differences, object positions and stacking patterns, so that the objects can be picked up precisely by the robot. This allows even irregular or mixed loads to be processed reliably.
3D Surface Inspection
Dents, deformations, assembly errors and height differences can be accurately detected with 3D surface inspection regardless of color or contrast. The three-dimensional measurement data is therefore suitable for automated quality control as well as for checking dimensional accuracy and completeness.
Fundamentals of 3D Imaging
3D sensors provide the measurement data as a 3D point cloud and can also output intensity values depending on the sensor model. Industrial 3D image processing uses 2D/3D profile sensors and snapshot sensors, among others.
2D/3D profile sensors project a laser line onto the measured object and record individual 2D height profiles. The relative movement between the sensor and object combines the sequentially captured profiles into a complete 3D point cloud. This method is particularly suitable for continuous measurement of moving objects, for example on conveyor belts or in production lines.
3D sensors capture the entire measuring volume in a single recording. To create a complete 3D point cloud, the object must not move during capture. This method is therefore particularly suitable for high-resolution measurements of static objects as well as for capturing the finest surface details.
Coordinate Systems for Processing 3D Data
Coordinate systems are of central importance for the spatial processing of 3D data. The measuring points recorded by the 3D sensor refer to the sensor’s coordinate system. The sensor coordinate system defines the position of each measurement point in relation to the position and orientation of the sensor and forms the basis for further processing of the depth data.
For robots, plants or image processing systems to use the collected data, it must be transferred to a global or plant-related coordinate system. This allows object positions, movements and gripping points to be clearly determined within the application.
Hand-Eye Calibration for Robot Vision
For robot vision applications, the coordinate systems of the 3D sensor and the robot must be linked to each other. The hand-eye calibration determines the geometric transformation between the two systems. This allows the robot to interpret the 3D positions recorded by the sensor in its own coordinate system and move to them in a targeted manner.
Depending on the application, a distinction is made between two variants:
Eye-to-Hand
In the eye-to-hand procedure, the sensor is permanently mounted in the working area and observes the robot’s working range. The calibration determines its position and orientation relative to the robot base.
Eye-in-Hand
With eye-in-hand, on the other hand, the sensor is located directly on the robot arm and moves with it. Its position relative to the tool center point (TCP) is determined here.
Calibration takes place via defined reference objects or calibration patterns, which are recorded by both the robot system and the 3D sensor. The transformation between the coordinate systems is calculated from the captured positions. The detected image coordinates can then be transferred from the sensor coordinate system to robot coordinates. Even small deviations in the calibration can lead to position errors in the gripper control or path planning.
Hand-Eye Calibration with M3DS Series Sensors
Various project and calibration examples are available on GitHub for integration into robotics and image processing systems. These include reference implementations for hand-eye calibrations and transformations of 3D coordinate systems. This allows typical integration and calibration processes to be traced and the examples provided can be used directly for your own application.
Working Distance and Measuring Range of 3D Sensors
Working Distance
Measuring Range
The measuring range of a triangulation-based 3D sensor is shaped like a pyramid stump. As the distance to the sensor increases, the measuring field in the x- and y-direction increases. This allows larger objects or scenes to be captured in the far range.
At the same time, the measuring points are spread over a larger area, so that the distance between the neighboring points increases. At close range, the point density of a 3D point cloud is therefore higher and fine structures can be captured in more detail.
The depth resolution in the z-direction also depends on the working distance. As the distance increases, the geometric sensitivity of the triangulation decreases, so that small height differences in the near range can be detected better.
How Is a 3D Point Cloud Created?
3D sensors use structured light to project light patterns onto an object. The deformations of these patterns give conclusions about the contour of the object. A camera then captures the deformation of the pattern on the surface. From their displacement, the sensor uses optical triangulation to calculate the depth information and generates a 3D point cloud.
Step 1 – Projection of a light pattern
The built-in light engine projects sequentially defined light patterns onto the surface of an object. The pattern serves as a defined reference structure for the subsequent depth calculation. wenglor’s 3D sensors use blue light, as its shorter wavelength enables higher optical precision and at the same time is less sensitive to ambient light. This allows even fine surface structures to be captured in a stable and high-contrast manner.
Step 2 – Deformation of the Light Pattern on the Object
If the structured light pattern hits an object, it is distorted spatially depending on its geometry. Differences in height, edges and indentations cause the projected stripes in the camera image to shift. The deformation of the pattern contains information about the shape and depth of the object.
Smooth, textured or highly reflective surfaces can affect the quality of the pattern image and thus also affect the subsequent depth calculation.
Step 3 – Camera Captures the Light Pattern
The camera then captures the projected light patterns from a specified angle. During capture, the system analyzes the exact position and deformation of the projected patterns in the image. A complete data set of the scene is created from several recorded individual images. How clearly the patterns can be seen in the camera image depends, among other things, on the exposure time, surface reflection, ambient light and optical resolution of the sensor.
Step 4 – Calculate the Depth Value by Triangulation
The camera and the projector view the measurement object from different angles. Together with a surface point, they form a geometric triangle. Since the positions of the camera and projector are known to each other, the sensor can calculate the depth value of each individual measurement point from the shift of the recorded patterns.
This process is called optical triangulation and forms the basis for the spatial reconstruction of the visible object surface.
Step 5 – Creating the Point Cloud
Based on the camera calibration, the sensor generates a 3D point cloud from the calculated depth values. Each measuring point is described by its x-, y- and z-coordinates and thus has a unique position in the space. This point cloud thus represents the surface structure of the object true to detail.
Further information such as intensity or so-called confidence values can also be stored. While intensity data provides information about the brightness or reflectance properties, the confidence value indicates the quality or reliability of an individual measurement point.
Processing 3D Data
In structured light projection, different reconstruction methods can be used to generate 3D data. The stripe method and the phase method differ in the type of projected patterns and their evaluation. Both methods use gray code sequences, which allow the projected light patterns to be clearly assigned to the captured image points. This assignment forms the basis for calculating the depth values. Which method is suitable for a measurement task depends, among other things, on the required recording time, the condition of the object surface and reflections or ambient light.
Stripe and Phase Method Comparison
Metodo a strisce
Nel metodo a strisce, sulla superficie dell’oggetto vengono proiettate strisce di luce ben delimitate e ad alto contrasto, di larghezza variabile. Grazie all’elevato contrasto, i motivi proiettati sono particolarmente ben visibili su superfici scure o altamente riflettenti. Allo stesso tempo, i modelli binari consentono tempi di illuminazione molto brevi, rendendo il metodo particolarmente adatto per superfici complesse.
Metodo a fasi
Il metodo a fasi prevede la proiezione sulla superficie dell’oggetto di schemi di strisce luminose sinusoidali con diversi sfasamenti. Invece di singole strisce, viene valutata la distribuzione continua della luminosità dei modelli proiettati. In questo modo è possibile determinare la posizione di un punto di misurazione con precisione subpixel, ottenendo dati 3D particolarmente uniformi, ad alta risoluzione e ricchi di dettagli.
Visualizzazione delle nuvole di punti 3D
Rappresentazione a falsi colori
Nella rappresentazione a falsi colori, ai valori di profondità della nuvola di punti 3D vengono assegnati diversi colori. Ciò consente di visualizzare le differenze di altezza e i contorni dell’oggetto, rendendo possibile una rapida interpretazione dei dati di misura.
Immagine di intensità
Oltre ai valori di profondità, il sensore è in grado di generare un’immagine di intensità. Si tratta di una classica immagine in scala di grigi che contiene informazioni sulle proprietà riflettenti della superficie.
Immagine di confidenza
L’immagine di confidenza mostra la qualità o l’affidabilità dei singoli punti di misurazione all’interno della nuvola di punti. In questo modo è possibile riconoscere e valutare in modo mirato aree con un calcolo incerto della profondità o una scarsa qualità del segnale.
Parametri di registrazione dei sensori 3D
Tempo di illuminazione
Guadagno
Sottocampionamento
Indicazioni di precisione secondo VDI/VDE
Stack Size
La Stack Size (dimensione dello stack) indica il numero di singole immagini o proiezioni campione utilizzate per un’acquisizione 3D. Una dimensione dello stack maggiore fornisce maggiori informazioni sull’immagine e può ridurre il rumore di misurazione. Allo stesso tempo, il tempo di registrazione si allunga, riducendo la velocità di registrazione massima raggiungibile.
Altre tecnologie 3D
Stereo passivo
Con questo metodo, due camere riprendono lo stesso oggetto con un’unica angolazione. Registrando gli stessi punti di riferimento con entrambe le camere, è possibile calcolare la posizione spaziale del punto di riferimento nello spazio in base allo spostamento delle immagini. La difficoltà qui consiste nell’identificare lo stesso punto con entrambe le camere. Ad esempio, se si osserva una superficie a basso contrasto, come una parete bianca, questo metodo non è ottimale.
Stereo attivo
La struttura corrisponde a quella dello stereo passivo. Inoltre, viene proiettato un modello (ad esempio punti distribuiti casualmente) sulla superficie. Ciò facilita l’assegnazione di un punto di entrambe le camere.
Tempo di volo
In questo metodo, la distanza tra l’oggetto e il sensore viene determinata in base al tempo di propagazione della luce. In questo caso, il sensore emette impulsi luminosi che colpiscono un oggetto e vengono da esso riflessi. La distanza viene determinata in base alla durata della riflessione degli impulsi luminosi. In questo modo è possibile determinare le informazioni sulla profondità, come le strutture o le distanze degli oggetti.
Triangolazione laser
Nella triangolazione laser, un laser proietta una linea sull’oggetto che viene rilevata da una camera con un angolo definito. Grazie allo spostamento della linea laser nell’immagine della camera è possibile determinare l’altezza della superficie dell’oggetto. Poiché viene acquisito un solo profilo alla volta, è necessario spostare l’oggetto o il sensore per generare dati 3D completi. Il procedimento è caratterizzato da un’elevata precisione, ma dipende da un movimento relativo definito durante l’acquisizione.
Comparison of Technologies
| Structured light | Passive stereo | Active stereo | Time of flight | |
|---|---|---|---|---|
| Resolution |  |
 |
|
 |
| Accuracy | |
|
|
|
| Ambient light | |
|
|
|
| Reading speed |  |
|
|
|
| Low-contrast objects | |
|
|
|
| Obstruction/shading | |
|
|
|
|
Resolution
|
|||
|---|---|---|---|
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Accuracy
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Ambient light
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Reading speed
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Low-contrast objects
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
|
Obstruction/shading
|
|||
|
Structured light
|
Passive stereo
|
Active stereo
|
Time of flight
|
Integration in Industrial Systems
For a 3D sensor to be used in industrial applications, it must be able to be seamlessly integrated into existing automation, robotics and image processing systems. In addition to the actual 3D data acquisition, standardized interfaces, software libraries and flexible integration options play a central role here. Modern 3D sensors offer extensive software and communication interfaces for this purpose, which enable easy integration into a wide range of system environments.
