@[toc](Contents) ## Summary ## We studied the discrimination of deformable objects by grasping them using 4 different robot hands / grippers: Barrett hand (3 fingers with adjustable configuration, 96 tactile, 8 position, 3 torque sensors), qb SoftHand (5 fingers, 1 motor, position and current feedback), and two industrial type parallel jaw grippers with position and effort feedback (Robotiq 2F-85 and OnRobot RG6). A set of 9 ordinary objects differing in size and stiffness and another highly challenging set of 20 polyurethane foams differing in material properties only was used. We systematically compare the grippers' performance, together with the effects of: (1) type of classifier (k-NN, SVM, LSTM) operating on raw time series or on features, (2) action parameters (grasping configuration and speed of squeezing), (3) contribution of sensory modalities. Classification results are complemented by visualization of the data using PCA. The dataset comprises 23820 unique measurements. Every measurement is a time series from all available sensory channels during one compression of an object. The dataset available complements the research article ([arxiv-pdf][1]). ## Objects set ## We considered two sets of deformable objects. The first set, *ordinary objects set*, consisted of 9 mostly cuboid objects with a different size and degree of deformability. The second set, *polyurethane foams set*, consisted of 20 polyurethane foam blocks of similar size, along with reference values for elasticity and density provided by the manufacturer. Cuboids are preferred over spheres, as the contact surface area does not change during deformation. For all the objects used, the deformation can be regarded as elastic---objects returning to their undeformed shapes once the external force is removed---although for some objects, this recovery was slow (e.g., Blue die and Blue cube). ### Ordinary objects set ### This set of 9 deformable objects is visualized below, approximately spread out by the stiffness/elasticity of the objects and their volume. The *yellowcube* is composed of the same material as the *yellowsponge*---it has been cut out from another exemplar of the same sponge, aiming at the dimension of the Kinova cube. The same is true for *bluedie* and *bluecube*. Conversely, *whitedie*, *kinova cube*, *yellowcube* and *bluecube* have roughly same dimensions but different material composition and hence stiffness. The dataset has been deliberately designed in this way, in order to test which of the object properties are key for model-free haptic object recognition.  ### Polyurethane foams set ### To complement the "ordinary objects set", we tested on another set of objects: 20 polyurethane foams. These were samples provided by a manufacturer of mattresses. Their labels encode also reference values for key physical properties: elasticity and density.  Overview of foams' properties: - Label – code from manufacturer. - Density is in kg · m −3 . - Elasticity is the CV 40 in kPa – “compression stress value at 40%”. (See “Polymeric materials, cellular flexible – Determination of stress-strain characteristics in compression, Part 1: Low-density materials,” 1986.)  ## Robot grippers and hands ## Four different devices were employed, with different morphology and hence degree of anthropomorphism (industrial type parallel-jaw grippers vs. robot hands with fingers), different options for controlling the squeezing action (compression speed and, if available, gripper configuration), and different type and dimension of feedback. Below we detail technical parameters of every gripper: the action parameters, the sensory channels, the stopping criterion of the grasping action and the sampling rate---all these may affect the discrimination of the object being compressed.  ### OnRobot RG6 ### The gripper has a 160 mm stroke and adjustable maximum force threshold. The gripper fingertips' surface area is 866 mm^2. - Action parameters: Force threshold and closing velocity. We used two force thresholds: 39 N and 109 N. This choice also affects the gripper closing speed. The thresholds were chosen such that the closing speed approximately corresponds to the speeds used on the Barrett Hand: v1=46 and v2=93 mm/s. - Sensory channels: 2 channels: gripper position (gap between jaws), motor current. - Stopping criterion: Force threshold reached. - Sampling rate: 15 Hz. An average sample from compression of a single object contained around 45 points in every channel (v1), 41 data points for v2. ### Robotiq 2F-85 ### The gripper has 2 fingers with 85 mm stroke and offers a grip force from 20 N to 235 N. The fingertips dimensions can be approximated to a rectangle of 37.5 x 22 mm. - Action parameters: Velocity control was used for the closing. This is quoted in % of the maximum closing speed. Four different nominal closing speeds were executed: 0.68% (v0.0068, approx. 1.6 mm/s), 20.69% (v0.21, 46 mm/s), 63.42% (v0.63, 80 mm/s), and 100% (v1.0, 131.33 mm/s). - Sensory channels: 2 channels: gripper position (gap between jaws 0--85 cm), motor current (A). - Stopping criterion: Position does not change anymore (empirical threshold: 8 samples). - Sampling rate: 100 Hz. An average sample from compression of a single object contained around 760 points in every channel (v0.0068), 439 (v0.21), 356 (v0.63), and 319 (v1.0) data points, respectively. ### qb SoftHand (Research edition) ### This is an anthropomorphic robotic hand with five fingers controlled via a single electric motor---synchronous opening or closing of all fingers. Thanks to passive compliance in the joints, it will conform to differently shaped objects. It can reach a grasp force of 62 N and has a nominal payload of 1.7 kg. - Action parameters: The hand configuration cannot be changed, but one can choose where to place the object. We considered two actions to minimize (action1 or a1) and maximize (action2 or a2) contact with the thumb. The hand is controlled by setting way points to the single motor. We achieved two closing velocities by setting the closing time to 2.5 s (v1) and 1.5 s (v2). - Sensory channels: 2 channels: motor position and motor current. Since the single motor drives all coupled fingers together, these values indirectly (and ambiguously) code for the position and effort between the fingers and the object. - Sampling rate: 10 Hz. An average sample from compression of a single object contained around 28 data points in every channel (v1), and 17 data points for v2. ### Barrett Hand ### The Barrett Hand (model BH8-282) has three fingers, of which two can rotate around the base. This design allows numerous configurations of the three fingers, with the only restriction that the spread joints of Finger 1 and 2 are mirroring each other. - Action parameters: We used two different finger configurations to press the object in the hand: opposing fingers (action1 or a1) and lateral configuration with all fingers on one side pressing against the palm (action2 or a2) (see Fig.~\ref{fig:all_hands}). Two objects, blue die and pink die were too big to be pressed in the lateral finger configuration. Velocity control was used with two different joint velocities: v1=0.6, and v2=1.2 rad/s. For action2 we used forward kinematics to calculate an approximation of the object compression velocity to be 46 and 93 mm/s, respectively. - Sensory channels: The hand is equipped with 96 pressure-sensitive cells (24 per fingertip and 24 on palm) and 3 fingertip torque sensors. In addition, 8 joint angles from encoders were recorded. - Sampling rate: All data was recorded at 200 Hz, but the effective rate of tactile sensor readings was around 25 Hz. An average sample from compression of a single object contained around 260 data points in every channel (v1), 220 data points for v2. ## Visualization of raw data The effect of action parameters and grasped object. *yellowsponge* and *yellowcube* are from the same material; *yellowcube* and *whitedie* have the same size.  ## Dataset overview ## The data we collected is available in the Files section, with the top level structure given by the gripper used, then the object type (ordinary objects vs. foams) and then by the action parameters. The dataset comprises 23820 unique measurements. Every measurement is a time series from all available sensory channels during one compression of an object. ## Models for Classification - Parameters **Performance Assessment of Classification Algorithms:** The performance of each classification algorithm utilized is contingent upon the careful selection of hyperparameters. Our approach involved tailoring hyperparameters to each specific dataset; for instance, when reporting performance on a particular action such as a1v0.6 for the Barrett Hand, it implies that distinct hyperparameters were optimized for that dataset. Conversely, when reporting on the 'all' category, a singular hyperparameter configuration was employed across all action types within the set. This strategy was not applied in our transfer learning studies. The k-Nearest Neighbors (k-NN) and Support Vector Machine (SVM) classifiers were developed using the SciKit-learn library. Hyperparameters were determined through a grid search with evaluation conducted on a validation set. The performance results we present are based on the test set, adhering to the established best practices of maintaining distinct training, validation, and test sets in machine learning research. 1) **k-Nearest Neighbors (k-NN):** The k-NN classifier was trained using either raw sensory data or hand-crafted features. The Euclidean metric was employed, and the optimal number of neighbors, K, was empirically determined by evaluating performance on the validation set. Consequently, each gripper was assigned an optimal K value for the object set it interacted with, whether objects or foams. 2) **Support Vector Machine (SVM):** For the SVM classifier, training was conducted on hand-crafted features. A grid search approach was utilized to ascertain the optimal hyperparameters (kernel type, C, and γ) specific to each gripper, action parameters, and object set. We evaluated three kernel types: linear, sigmoid, and Radial Basis Function (RBF). The parameter C was selected from a logarithmic distribution spanning \(10^{-5}\) to \(10^{-3}\), exploring 7-10 samples within this range. Similarly, γ values were chosen from a logarithmically spaced set within the same bounds when relevant, considering 7-10 samples. The combination that yielded the best validation performance was chosen as the optimal hyperparameter set. The Long Short-Term Memory (LSTM) network was adapted from an existing model available in the PyTorch library. Training was executed using the Adam optimizer, with hyperparameters varied across several dimensions: the number of LSTM layers (2 or 4), the size of the hidden layer (32, 64, 128, or 256), and the learning rate, which was logarithmically distributed between \(10^{-5}\) and \(10^{-3}\). Batch sizes were empirically determined to fit within the constraints of the available memory resources. Interestingly, while some datasets allowed for the use of all training data in a single batch, this did not necessarily correlate with improved accuracy. An exhaustive random search was employed to traverse the hyperparameter space. Computations were performed on a robust system configuration comprising 16 cores/32 threads, 256GB RAM, 500GB SSD, and 8 NVIDIA GTX 1080Ti GPUs. Models that demonstrated superior performance metrics on the test set were selected as optimal. In cases where multiple models exhibited equivalent performance, the model with the most stable learning curve was preferred. For selected hyperparameters, see Table 1.  ## Visualizations using PCA ## In order to understand what the dominant sources of variability in the data are---the different objects explored, the way they are manipulated, the sensory channels used---we employed an unsupervised learning techniques, specifically a standard dimensionality reduction technique Principal Component Analysis (PCA). In the visualizations shown below, different objects are shown with different colors and different action parameters with different markers. ### OnRobot RG6 ### For the OnRobot RG6 gripper, the main clusters are formed by the action parameters (force threshold / squeezing speed). The second dimension is due to differences between objects. The softer objects---yellow cube, yellow sponge, blue cube, and blue die---are clustered together despite their different dimensions, indicating that stiffness is the main source of variability.  Fig. caption. OnRobot RG6 - objects - Visualization using PCA. Colors - objects. Markers - action parameters.  Fig. caption. OnRobot RG6 - foams - Visualization using PCA. Colors - objects. Markers - action parameters. ### Robotiq 2F-85 ### On the Robotiq 2F-85 gripper and the objects set, the first PC is dominated by the compression velocity, with the two slow speeds grouped together. The second PC is then taken up by the object characteristics, dominated by their stiffness.  Fig. caption. Robotiq 2F-85 - objects - Visualization using PCA. Colors - objects. Markers - action parameters. On the foams dataset, the velocity dominates forming the principal clusters. Note also that the slower speeds better separate the foams.  Fig. caption. Robotiq 2F-85 - foams - Visualization using PCA. Colors - objects. Markers - action parameters. ### qb Soft Hand ### On the qb Soft Hand, two different configurations and two different velocities were tried. The first principal component is dominated by the velocity parameter, with smaller velocity *v1* on the left. The second component is dominated by the objects and in particular their volume.  Fig. caption. qb Soft Hand - objects - Visualization using PCA. Colors - objects. Markers - action parameters. ### Barrett Hand ### On the Barrett Hand and the objects set, the first PC is taken mainly by the grasping configuration with the opposing fingers (*a1*) on the left side. Material elasticity seems to dominate PC2 with the softer objects at the bottom. The compression velocity seems to be the smallest source of variability. Similar conclusions can be drawn from the visualizations of the foams dataset further below.  Fig. caption. Barrett Hand - objects - Visualization using PCA. Colors - objects. Markers - action parameters.  Fig. caption. Barrett Hand - foams - Visualization using PCA. Colors - objects. Markers - action parameters. ### Variability across grippers ### The same analysis was performed on also across multiple grippers to assess the effect of the gripper morphology on the variability in the sensory data. As expected, the different device form the principal clusters. This poses a challenge for any classifiers that should work on different robot hands/grippers. Only the devices with two sensory channels - OnRobot RG6, Robotiq 2F-85, and qb Soft Hand - could be directly compared. On the objects set, qb Soft Hand forms a compact cluster on the center left; OnRobot RG6 on the top. The Robotiq 2F-85 data points are most scattered in the bottom part of the plot.  Fig. caption. 3 grippers - objects - Visualization using PCA. Colors - objects. Markers - grippers and action parameters. On the foams dataset, we analyzed the OnRobot RG6 and Robotiq 2F-85 grippers together. The first principal component separates the two parallel jaw grippers. Interestingly, the second component separates the closing speeds, preserving their order.  Fig. caption. OnRobot RG6 and Robotiq 2F-85 - foams - Visualization using PCA. Colors - objects. Markers - grippers and action parameters. ## Visualizations using PCA and t-SNE - interactive tool ## The plots in the section above are only a small selection of possible visualizations. In order to allow for additional analyses, we make available an interactive tool that applies either PCA or t-SNE algorithm over the feature space and visualizes it in 2D. For each gripper, a combination of object set, action, and sensor can be set. This allows empirical investigation of, among other things, the contribution of individual sensors and the relationships between individual actions. The legend is generated automatically. In the case of an object set, the color encodes the measured object, whereas a lighter color encodes a known object of the same material but smaller. In the case of foam set, color codes the density and, in the case of the same density, the hardness. Each action has its own mark, and it is possible to find out what is the main factor in the appearance of the clusters. On top, when moving the mouse, the properties of the closest object are logged in the console, and it is possible to see if, for example, the series in which the measurement was made has an influence. It is also possible to click on a marker and display a graph of the corresponding measurement for even better insight. The interactive tool can be found here under *files/unsupervised-analysis-interactive-tool* [(link to .zip for download)][2]. For each gripper, a file with data and script exists. To run the interactive tool, just run the interactive *unsupervised analysis* python script.  ## Publication and video ## This video accompanies the article: Pliska, M.; Patni, S. P.; Mares, M.; Stoudek, P.; Straka, Z.; Stepanova, K. & Hoffmann, M. (2024), 'Single-grasp deformable object classification: the effect of gripper morphology, sensing modalities, and action parameters', IEEE Transactions on Robotics 40, pp. 4414 - 4426. [https://doi.org/10.1109/TRO.2024.3463402](https://doi.org/10.1109/TRO.2024.3463402) Available also at: [https://arxiv.org/abs/2204.06343](https://arxiv.org/abs/2204.06343) Accompanying video is at: [https://youtu.be/-6cmQrSzbCs](https://youtu.be/-6cmQrSzbCs) ## Acknowledgements ## This work was co-funded by the European Union under the ROBOPROX (Robotics and advanced industrial production) project (reg. no. CZ.02.01.01/00/22_008/0004590). K.S. was additionally supported by the Czech Science Foundation (project no. GA21-31000S). The work originated in the IPALM project (H2020-FET-ERA-NET Cofund-CHIST-ERA III / TAČR EPSILON, no. TH05020001). [1]: https://arxiv.org/abs/2204.06343 [2]: https://files.de-1.osf.io/v1/resources/zetg3/providers/osfstorage/632b01d36c24010c30509a0a/?zip=