The automated reconstruction of Building Information Modeling (BIM) objects from point cloud data is still ongoing research. A key aspect is retrieving the proper observations for each object. After segmenting and classifying the initial point cloud, the labeled segments should be clustered according to their respective objects. However, this procedure is challenging due to noise, occlusions and the associativity between neighboring objects. This is especially important for wall geometry as it forms the basis for further BIM reconstruction. In this work, a method is presented to automatically group wall segments derived from point clouds according to the proper walls of a building. More specifically, a Conditional Random Field is employed that evaluates the context of each wall segment in order to determine which wall it belongs too. First, a set of classified planar primitives is obtained trough algorithms developed in prior work. Next, both local and contextual features are extracted based on the nearest neighbors and a number of seeds that are heuristically determined. The final wall clusters are then computed by decoding the graph. The experiments prove that the proposed method is a promising framework for wall clustering from unstructured point cloud data. Compared to a conventional region growing method, the proposed method reduces the rate of false positives, resulting in better wall clusters. A key advantage of the proposed method is its capability of dealing with complex wall geometry in entire buildings opposed to the presented methods in current literature.