Optical High-Speed Rolling Mark Detection Using Object Detection and Levenshtein Distance

Featured Application

Railroad Infrastructure Detection.

Abstract

This paper presents an automated high-speed rolling mark recognition system for railroad rails utilizing image processing techniques. Rolling marks, which consist of numbers, letters, and special characters, were engraved into the rail web as 3D information. These rolling marks provide crucial details regarding the rail manufacturer, steel quality, year of production, and rail profile. As a result, they empower rail infrastructure managers to gain valuable insights into their infrastructure. The rolling marks were captured using a standard color camera under dark field illumination. The recognition of individual numbers, letters, and special characters was achieved through state-of-the-art deep neural network object detection, specifically employing the YOLO architecture. By leveraging reference rolling marks, the detected characters can then be accurately interpreted and corrected. This correction process involves calculating a weighted Levenshtein distance, ensuring that the system can identify and rectify partially misidentified rolling marks. Through the proposed system, the accurate and reliable identification of rolling marks was achieved, even in cases in which there were partial errors in the detection process. This novel system thus has the potential to substantially improve the management and maintenance of railroad infrastructure.

1. Introduction

In the European Union, the extensive railway network spans approximately 216,000 km [1]. To ensure proper availability, ongoing maintenance and servicing are essential. Currently, manual track inspections form a significant part of these maintenance activities. For instance, in the “Deutsche Bahn” network (approximately 38,400 km), track inspections are scheduled at least every two years. In addition to manual inspections, there are also advanced maintenance and servicing systems equipped with specialized sensors. These systems employ sensors mounted on traction units to detect and identify railway-related objects within the track network.

These comprehensive systems provide technicians and managers with a holistic overview of track conditions. This system is instrumental in planning the necessary actions and interventions. It combines various sensory inputs to offer valuable insights into track conditions, aiding in decision-making processes. The majority of railway tracks installed worldwide consist of Vignole rails, which can be categorized into three different parts—the rail head, rail web, and rail foot—as depicted in Figure 1.

Typically, the rolling mark is positioned on the rail web at consistent intervals along the rail line, as illustrated in Figure 2. This mark serves as an identifier and contains essential information pertaining to the rail. It comprises a combination of letters, numbers, and special characters, often including horizontal lines as well. These characters collectively convey specific details about the rail, allowing for identification and data interpretation.

In this paper, we introduce a novel camera-based system designed for automatically extracting information from rolling marks. The paper is organized as follows.

Section 2 provides an in-depth look into rolling marks. Section 3 describes the method for detecting rolling marks, including the physical principle of measurement and the basic principles of the evaluation software. In Section 4, we demonstrate an exemplary evaluation of the presented method. We use test data to highlight the strengths and weaknesses of the method. Section 5 summarizes the paper, and Section 6 provides an outlook on potential further applications of the system.

2. Rolling Marks

For Vignole rails of contemporary design weighing 46 kg/m or above, the DIN EN 13674-1 standard provides comprehensive guidelines for both conventional and high-speed railway lines. Within this standard, specific requirements are outlined for the textual composition of rolling marks, as depicted in Figure 2. These specifications encompass the following aspects:

• Rolling mill;
• Steel grade;
• Rolling year;
• Rail profile.

Each rolling mark is carefully rolled into the rail web at intervals no greater than 4 m, ensuring that the marks are prominently raised within the range of 0.6 mm to 1.3 mm. The characters comprising the rolling mark exhibit a consistent height falling between 20 mm and 25 mm. These specifications have been in effect since November 2002, serving as a standardized framework for rolling mark implementation. Figure 2 visually portrays a representative rolling mark adhering to the aforementioned structural guidelines. In this specific example, the encoded information contained within the rolling mark was as follows:

• The rail was rolled in the Donawitz rolling mill (Donawitz Steelworks is a steel mill located in Donawitz near Leoben in Styria/Austria; it is particularly well known for the first application of the Linz-Donawitz process (LD) for steel production).
• The quality of the steel corresponds to grade R 350 HT.
• The rolling year was 2012.
• It was a rail of profile 54E2.

Rails manufactured prior to 2002 often deviate from the standard convention described earlier, particularly with regard to the placement of the steel grade indication at the right end of the rolling mark. Additionally, these rolling marks typically include the specification of the month of production and utilize an older profile designation, such as the usage of “UIC 54E” instead of the “54E2” designation.

The accurate and automated localization of these specific markings holds significant importance for rail infrastructure operators, as it allows for targeted insights into the rail infrastructure. This capability facilitates the effective planning of maintenance actions according to the age of the rails. Furthermore, it contributes to enhanced safety by enabling the identification of older rails that require replacement in a precise and targeted manner.

3. Camera-Based Rolling Marks Detection

A rolling mark can be considered a “surface anomaly” due to its distinctive elevation on the rail. To enhance the visibility of these marks, dark field illumination is employed in image processing techniques. In this work, the illumination setup was similar to that depicted in Figure 3. All objects were located within the structure gauge specified in the railroad area. The emitted light was directed onto the rail surface but not directly back into the camera. Only in the vicinity of a raised surface, such as a rolling mark, does some of the light reflect back to the camera, resulting in brighter contours of the mark in the captured image (as shown in Figure 4).

Furthermore, the illumination system differentiated between red and green colors, exploiting the photometric stereo effect. This technique takes advantage of the varying angles of light incidence to create shading variations, which can help to accentuate surface features. A similar principle was utilized in [2] to detect defects on the rail surface. Building upon this principle, the present work employed it to further enhance the visibility of rolling marks. Furthermore, an optical bandpass filter, tuned to the wavelength of the illumination, was used, reducing ambient influences, such as sunlight.

To enable the extraction of information from a rolling mark, the process involved two main steps: detection (localization) of the mark in the image and subsequent interpretation. While conventional computer vision approaches often rely on a combination of neural networks for object detection, such as You Only Look Once (YOLO) [3], along with additional text recognition software (OCR, Optical Character Recognition, e.g., [4,5]), this work adopted a different approach (YOLO is a widely used object detector that has been intensively characterized in numerous computer vision benchmarks [6] offering high frame rates and a relatively user-friendly software framework).

In this work, each individual character of the rolling mark string was detected separately using the deep learning-based YOLO object detector. This approach allows for more precise localization of each character within the mark. Subsequently, the identified characters were passed to a parser that utilized the Levenshtein distance metric [7] for the interpretation and analysis of the rolling mark. By treating each character as an individual object of interest, the proposed approach provides more flexibility and accuracy in capturing the intricate details of the rolling mark. This method avoids the need for additional preprocessing steps, such as converting the region of interest to a binary image, which are commonly employed in traditional object detection and OCR techniques. The parser, leveraging the Levenshtein distance metric (explained in detail in Section 3.2), compared the detected characters with reference rolling marks and performed corrective actions according to the calculated distance. This enabled the system to handle partially incorrectly recognized marks and ensure the accurate interpretation of the rolling mark content. This novel combination of an object detection technique and the power of the Levenshtein distance-based parser provides a robust and efficient solution for rolling mark detection and interpretation.

3.1. Parser Based on a Weighted Levenshtein Distance

The basic idea is to create reference character strings that are generally possible in a rolling mark. Then, the corresponding Levenshtein distance (a distance metric between two strings) is calculated between a reference and a detected rolling mark in the camera image. Finally, a range search algorithm [8] is used to determine the correct sequence of rolling mark characters. For each calculation of the Levenshtein distance, the backtrace between the two strings under consideration is also determined. This backtrace then yields the “path” between two words. The associated path corresponds to the sequence of necessary change steps to move from the detected word to the reference. From this, the characters that are incorrectly recognized can be deduced. Thus, the corrected rolling mark is compiled from a sequence of instructions consisting of documented insertion, deletion, and replacement processes of characters, which will be described in detail in the next section.

3.2. Levenshtein-Distance

The simple Levenshtein distance (edit distance) between two strings $s$ and $t$ is defined as the cost of necessary operations to transfer $s$ to $t$ [9]. It can be calculated as ${\delta }_{lev}\left(s,t\right)=L_{\left|s\right|,\left|t\right|}$, where the following recursion equation holds for $L$:

$$\begin{gathered} \hspace{1em}{L}_{\mathrm{0,0}}=0 \\ {L}_{i,j}=\mathrm{m}\mathrm{i}\mathrm{n}=\left\{\begin{array}{c}{L}_{i-1,j-1}+0, \hspace{1em}keep\\ L_{i-1,j-1}+1,\hspace{1em} exchange\\ L_{i-1,j}+1,\hspace{1em}\hspace{1em} insert\hfill \\ L_{i,j-1}+1,\hspace{1em}\hspace{1em} delete\hfill \end{array}\right. \end{gathered}$$

(1)

3.3. Weighted Levenshtein Distance

The Levenshtein distance, a useful metric for measuring the similarity between two strings, assumes equal costs for insertion, deletion, and substitution operations. However, in the case of correcting rolling marks, it is often necessary to assign different costs to these operations on the basis of their specific impact on the accuracy of the interpretation.

To address this, the Levenshtein distance can be extended with variable costs by defining functions to calculate the costs for exchanging, inserting, and deleting characters. For example, the function $d_{exchange}(s\left[i\right],t\left[j\right])$ can be used to determine the costs incurred when swapping the characters $s\left[i\right]$ and $t\left[j\right]$. Similarly, the functions $d_{insert}\left(t\left[j\right]\right)$ and $d_{delete}\left(s\left[j\right]\right)$ can be defined to calculate the costs for inserting character $t\left[j\right]$ or deleting character $s\left[j\right]$, respectively.

These cost functions can be implemented in various ways. One approach is to use a static table that assigns predetermined costs for each operation. Alternatively, the costs can be adjusted dynamically during runtime on the basis of the properties of the rolling mark data, such as the significance of certain characters or the likelihood of specific errors.

$$\begin{gathered} \hspace{1em}{L}_{\mathrm{0,0}}=0 \\ {L}_{i,j}=\mathrm{m}\mathrm{i}\mathrm{n}=\left\{\begin{array}{ll}{L}_{i-1,j-1}+0,& keep\\ L_{i-1,j-1}+d_{exchange}(s\left[i\right],t\left[j\right]),& exchange\\ L_{i-1,j}+d_{insert}\left(t\left[j\right]\right),& insert\\ L_{i,j-1}+d_{delete}\left(s\left[j\right]\right),& delete\end{array}\right. \end{gathered}$$

(2)

Table 1. Results table including backtrace for a weighted Levenshtein distance between the German words $s=\mathrm{“}ITALIEN\mathrm{”}$ and $t=\mathrm{“}ALLEIN\mathrm{”}$. The orange coloured fields indicate the path between the two words.

Exchangeor Keep	← Delete →
← Insert →		„“	I	T	A	L	I	E	N
	„“	0.0	0.9	1.8	2.7	3.6	4.5	5.4	6.3
	A	1.0	1.8	2.7	1.8	2.8	3.8	4.8	5.8
	L	2.0	2.4	3.2	2.7	1.8	2.8	3.8	4.8
	L	3.0	3.3	3.8	3.6	2.7	3.2	4.2	5.2
	E	4.0	4.2	4.7	4.5	3.6	4.1	3.2	4.2
	I	5.0	4.0	5.0	5.4	4.5	3.6	4.1	5.1
	N	6.0	4.9	5.9	6.3	5.4	4.5	5.0	4.1

shows the results obtained for the two German words $s=\mathrm{“}ITALIEN\mathrm{”}$ and $t=\mathrm{“}ALLEIN\mathrm{”}$ at costs:

$$\begin{gathered} d_{insert}\left(t\left[j\right]\right)=1 \\ {d}_{delete}\left(s\left[j\right]\right)=0.9 \\ {d}_{exchange}\left(s\left[i\right],t\left[j\right]\right)=0.9+\alpha \end{gathered}$$

(3)

with $\alpha =\left\{\begin{array}{ll}\mathrm{0,5},& s\left[i\right]\in \{“\mathrm{I}”,”\mathrm{T}”,”\mathrm{L}”\},t\left[j\right]\in \{“\mathrm{I}”,”\mathrm{T}”,”\mathrm{L}”\},s\left[i\right]\ne t\left[j\right]\\ 1,& other\end{array}\right.$.

Starting with the Levenshtein algorithm, the cost matrix $L$ is initialized with $L_{0,0}=0$, and the matrix is gradually filled in for each position (i, j) according to the comparison of characters from the input strings s and t. The cost values increase as the algorithm progresses, ultimately resulting in the value at $L_{\left|s\right|,\left|t\right|}$, which represents the total cost or distance between the two strings.

In the case of the weighted Levenshtein distance ${{\delta }_{lev}}^i\left(s,t\right)=L_{\left|s\right|,\left|t\right|}$, the value at $L_{\left|s\right|,\left|t\right|}$ is 4.1. The weights associated with each operation (insertion, deletion, and exchange) contribute to this calculated distance. By considering these weights, the algorithm assigns different costs to each operation, reflecting the specific requirements of the problem at hand.

Additionally, the path or backtrace between the two strings can be determined on the basis of the entries in the calculated Levenshtein matrix. This backtrace represents the sequence of correction steps needed to transform one string into the other, typically referred to as the edit operations. If we view the entries in the cost matrix as an “elevation profile”, with higher values representing higher elevations, the path along the valley in this profile corresponds to the optimal sequence of correction steps. Traversing this path in the matrix provides the corrected order of operations between the two strings. This reads:

• Keep $„N“$;
• Insert $„I“$;
• Keep $„E“$;
• Exchange $„I“$ with $„L“$;
• Keep $„L“$;
• Keep $„A“$;
• Delete $„T“$;
• Delete $„I“$.

3.4. Creating and Parsing with Reference Rolling Marks

By combining all known manufacturer designations, special characters for steel quality and manufacturing process, profile designations, year numbers, and months, it would be possible to generate an extensive list of reference rolling marks. However, due to the sheer number of elements involved, this list would be exceedingly large. For instance, a rolling mark conforming to the current standard would appear 100 times in the list, each with a different year. Hence, it becomes logical to generate a reference rolling mark that incorporates a year placeholder. This approach is justified by the fact that the manufacturer designation remains constant across rails from different years, while the manufacturing date varies. Additionally, the type designation itself is limited to a finite number of variants. For example, a rolling mark of the current standard would be “DO  12 54E2” (see Figure 2). The reference rolling mark is formed using the structure “Producer _ Quality _ Y-T Y-O _ Type”, where Y-T is a placeholder for the tens digit of the year and Y-O is a placeholder for the one’s digit.

3.5. Parsing with the Aid of Reference Rolling Marks

Utilizing each known textual structure, a comprehensive reference rolling mark is generated by considering all conceivable combinations of manufacturers, steel qualities, months, manufacturers, and type designations. Placeholder variables are included specifically for the years, as well as for the spaces between each distinct specification. Employing an efficient calculation method, such as tries [8], the weighted Levenshtein distance of the identified characters is computed.

The YOLO neural network, employed for detecting the distinctive characters of a rolling mark, can undergo preliminary testing. To achieve this, data that has not been processed by the neural network yet is utilized. The outcomes from the object detections are then compared with the actual values and meticulously documented in a table, known as the confusion matrix. This matrix demonstrates the neural network’s performance in terms of accurately identifying characters (referred to as True Positive—TP), instances in which confusion occurs, undetected characters (False Negative—FN), and instances in which characters are detected despite their absence (False Positive—FP). On the basis of the confusion matrix, tailored costs can be assigned to the network. Figure 5 depicts a confusion matrix consisting of 65 classes, encompassing 10 digits (0–9), 26 letters (A–Z), 26 special characters, a hyphen, a dot, a comma, and the background. Each entry in the matrix is normalized relative to the columns. The bluer a field appears, the closer its numerical value is to 1, while a white field corresponds to a numerical value of 0.

The costs for the weighted Levenshtein distance are chosen as follows: characters that are often confused, falsely detected, or not found at all according to the confusion matrix are easier to correct in the subsequent distance calculation. Therefore, the following considerations are made:

If a class is consistently detected in nearly every instance within the image (FN ≈ 0), the cost of an insert operation is consequently assigned a higher weight. Conversely, for a class that is infrequently detected (FN ≈ 1), the weight assigned to the insert operation is lower.

Likewise, the cost of deleting a character can be ascertained by examining the false-positive (FP) values in the confusion matrix. The occurrence of predicting a character class that does not actually exist is reflected in the FP values. If certain classes are more prone to such occurrences, the cost of deleting those characters is set to a lower value. Additionally, the confidence value of the predicted class c serves as a measure of “reliability”. For instance, if a character is partially obscured or unclear in the image, a lower confidence value is typically assigned.

In reality, there is a higher likelihood of confusion between the number “0” and the letter “O” compared with, for example, the letter “E”. The confusion matrix provides insights into which classes are more prone to such confusion (TP = 0), as well as classes that are rarely confused (TP ≈ 1, FP ≈ 0). When there is a high probability of confusion, the cost associated with swapping two characters should be relatively low. It is worth noting that swapping the two characters $s\left[i\right]$ with $t\left[j\right]$ can also be interpreted as deleting $t\left[j\right]$ and inserting $s\left[i\right]$ in place of $t\left[j\right]$. Therefore, the cost must be quantified as higher than a single operation to account for this combined action.

To address the potential bias toward shorter strings, the calculated weighted Levenshtein distance is further normalized by dividing it by the sum of the deletion costs ${\mathit{L}}_{0,\left|t\right|}$. This normalization ensures that shorter strings do not dominate the distance calculation, thus enabling the inclusion of longer rolling marks in range searches. Consequently, a specific weighted Levenshtein distance measure $\widehat{{{\delta }_{lev}}^i}\left(s,t\right)=\frac{{\mathit{L}}_{\left|s\right|,\left|t\right|}}{{\mathit{L}}_{0,\left|t\right|}}$ is employed to account for this normalization.

4. Example Evaluation and Results

To conduct an illustrative evaluation, a series of test recordings were conducted on the Wels–Passau railroad line, specifically on the section between Wels (route kilometer 0) and Riedau (route kilometer 42.2). This section corresponds to route number ÖBB 205 01 and route number DB 5831. The evaluation employed a dataset comprising 11,800 images capturing rolling marks. These images were captured using a measuring vehicle provided by the company Plasser & Theurer. A new image was captured every 250 mm, depicting a track section of approximately 340 mm.

4.1. Object Detection Results (Single Rolling Mark Characters)

Table 2. Number of individual classes in the data set.

Label	Count
0	6682 (10.4%)	L	35 (0.1%)	<06>	3 (<0.1%)
1	5011 (7.8%)	M	157 (0.2%)	<07>	5344 (8.3%)
2	601 (0.9%)	N	219 (0.3%)	<08>	19 (<0.1%)
3	173 (0.3%)	O	5579 (8.7%)	<09>	-
4	1233 (1.9%)	P	15 (<0.1%)	<10>	19 (<0.1%)
5	1503 (2.3%)	Q	-	<11>	-
6	5373 (8.4%)	R	273 (0.4%)	<12>	-
7	101 (0.2%)	S	491 (0.8%)	<13>	2 (<0.1%)
8	974 (1.5%)	T	246 (0.4%)	<14>	-
9	3092 (4.8%)	U	2929 (4.6%)	<15>	-
A	312 (0.5%)	V	2211 (3.4%)	<16>	-
B	83 (0.1%)	W	226 (0.4%)	<17>	-
C	2985 (4.7%)	X	841 (1.3%)	<18>	-
D	5235 (8.2%)	Y	284 (0.4%)	<19>	-
E	2589 (4.0%)	Z	221 (0.3%)	<20>	-
F	-	<00>	-	<21>	-
G	19 (<0.1%)	<01>	184 (0.3%)	<22>	-
H	283 (0.4%)	<02>	4 (<0.1%)	<23>	-
I	7374 (11.5%)	<03>	-	<24>	-
J	-	<04>	934 (1.5%)	<25>	-
K	17 (<0.1%)	<05>	241 (0.4%)	.	4 (<0.1%)
				–	69 (0.1%)

provides a comprehensive overview of the distribution of individual classes, while

Table 3. Graphical representation of the occurring special characters.

Label	Special Character
<01>
<02>
<04>
<05>
<06>
<07>
<08>
<10>
<13>

visually presents the special characters. The dataset was divided into three subsets: 80% for training data and 10% each for validation and test data. Special attention was given to ensuring that the test data included images from successive exposures, which are crucial for evaluating the subsequent analysis with the parser.

The training process involved an initial training phase, which consisted of 300 epochs with a batch size of 32 and an image size of 640 × 640 pixels. Subsequently, fine-tuning was conducted with half the batch size and a reduced learning rate until no further improvement in training performance could be observed. The training process made use of the hardware acceleration provided by 8 NVIDIA RTX A6000 GPUs.

Figure 6 shows the obtained results based on the common evaluation metrics used in the field of image classification and object recognition, namely the mAP (mean average precision), F1-score, precision, and recall [10]. The first column shows the precision averaged over all classes at a confidence value of 0.5. The second column shows the highest F1 score. Column three shows the maximum precision. This occurs at the confidence value given in the fourth column, and the fifth column indicates the recall at a confidence value of 0.

4.2. Measurement Speed Considerations

The image acquisition should take place at a driving speed of at least 80 km/h so that no railroad line closure is required. The camera and lighting must be designed accordingly. With a selected distance of 25 cm between two image captures, a frame rate of

$$\frac{80·\frac{\mathrm{km}}{\mathrm{h}}}{0.25·\mathrm{m}}=88.8\frac{1}{\mathrm{s}}$$

is required. This is possible with commercially available cameras and lighting. The bottleneck represents the data stream, which has to be processed at correspondingly high speeds. In the example evaluation, about $30\frac{\mathrm{GB}}{\mathrm{km}}$ were recorded. If the task was to evaluate the images directly and store only the detected rolling marks (without the original image data), the detection network needed to be accordingly fast. With an image size of 640 × 640 pixels, the average processing time of the YOLO detector in our case was 8 ms per image (7.4 ms for inference + 0.6 ms for non-maximum suppression). This implied a final measurement speed of:

$$\frac{0.25·\mathrm{m}}{0.008·\mathrm{s}}=31.25\frac{\mathrm{m}}{\mathrm{s}}=112.5\frac{\mathrm{km}}{\mathrm{h}}\widehat{=}8\frac{\mathrm{rolling} \mathrm{marks}}{\mathrm{s}}$$

4.3. Results of the Parser and Rolling Mark Correction and Application Scenarios

Defining a metric to determine whether the input data (detections from the neural network) have been correctly corrected or parsed is not a trivial task. Various factors can complicate the process of “correct correction” or even render it impossible. For instance, if a significant portion of a rolling mark is not visible, as illustrated in Figure 9, achieving a precise correction becomes challenging. Furthermore, attempting to establish a specific criterion, such as “the maximum allowable number of incorrectly detected characters”, in order to deduce the correct rolling mark is impractical due to the diverse and similar nature of rolling marks. A universal answer cannot be provided.

Moreover, it is important to note that without any correction, the input data already exhibit high quality with minimal errors. The need for correction primarily arises in exceptional cases. Consequently, the role of the parser often revolves around exclusive parsing rather than extensive correction. In general, it has been observed that the necessity for correction diminishes as the input data quality improves.

In light of these considerations, this section aims to provide insights into the expected quality of the results by showcasing example images and highlighting scenarios in which erroneous outcomes may arise.

It is feasible to generate different representations of rolling marks automatically. In Figure 7, each distinct rolling mark is assigned a defined color. In Figure 8, the color selection is based on the steel grade. However, it is also possible to filter for other information contained within the rolling marks. For instance, one can filter for rails from a specific manufacturer with a designated year of manufacture.

By examining Figure 7 and Figure 8, certain assumptions can be made: Initially, there were only two rails in the depicted area, represented by the light blue and green lines. Subsequently, a track with a higher steel grade (yellow line) was inserted. However, this track required multiple corrections, as indicated by the red, blue, and purple lines. It is possible that this track belongs to a batch with quality issues and may need to be replaced.

The light blue and green lines correspond to relatively recent tracks manufactured in 2011 and 2015, respectively, while the yellow line represents a much older track produced in 1998. The rails used for track correction (red, blue, and purple lines) are also younger, originating from the years 2010, 2014, and 2019. It is advisable for a track infrastructure operator to pay close attention to the track from 1998 (yellow line) during the next maintenance cycle, as further repairs may be necessary.

In Figure 9, the indication of the production month (originally September, represented by the Roman numeral IX) is obscured by a ground wire. Consequently, after parsing, the production month was incorrectly identified as January (represented by the Roman numeral I). Such “simple errors” can be automatically corrected by a multi-step correction procedure. In this process, all rolling marks that are located between two rail joints (welds, connecting lugs) are combined, and an additional correction process is performed. It is important to note that this analysis is based on test data only. None of the images were previously processed by the neural network.

5. Summary and Conclusions

This study focused on the recognition of rolling marks on railroad rails at high process speeds. Rolling marks are 3D lettering rolled into the rail web, comprising a distinctive sequence of characters. They convey crucial information about the rail manufacturer, production year, steel grade, and rail profile.

Images were captured using a color camera with dark field illumination. The YOLO neural network was employed to detect the characters present in the rolling marks. Leveraging the inherent structure of rolling marks, it was possible to infer the provided information. A weighted Levenshtein distance was computed between the detected characters and predefined reference rolling marks. This approach enables corrective measures to be applied to inaccurately detected characters. Since this method allows for the interpretation of all rail information, diverse analyses can be conducted.

For instance, it is feasible to automatically search for specific manufacturers or rolling marks from particular manufacturing periods. Additionally, transitions between different steel grades or rail profiles can be visualized. As shown in Section 4.3, potentially safety-critical areas can thus be found. Frequent repair of a track indicates a track with low quality. This provides rail infrastructure managers with a valuable tool for maintenance and the planning of maintenance intervals for their infrastructure.

Clear practical limitations of the proposed approach are heavy surface abrasion and overlap by vegetation or engineered rail fasteners (see Figure 9), i.e., situations in which the rolling mark structure is simply no longer present in the image. In general, however, the proposed approach is surprisingly robust, due to:

• A special camera illumination technique (dark field illumination);
• The use of optical bandpass filters for improved and thus more reliable image acquisition;
• A specially trained object detection algorithm (YOLO) for finding single rolling characters, which is not based on classical OCR (“Optical Character Recognition”) techniques;
• A subsequent text correction, which improves reliable recognition in the case of partially incorrect detections.

6. Outlook

6.1. Semi-Supervised Learning

The presented Levenshtein-based text correction technique introduces additional opportunities for automation. By leveraging the known correction steps for faulty detections, automatic correction of labels becomes feasible. These corrected labels can then be reintroduced into the training cycle of the neural network. As the system learns from this process, the combination of the neural network and correction algorithm can be considered a form of semi-supervised learning.

However, the Levenshtein-based correction method may not be effective for characters that have multiple valid variants, such as the steel grade and manufacturing time. In such cases, corrections can be performed on the basis of neighboring rolling marks. On the other hand, the manufacturer’s designation and profile designation fall into the category of “easy to correct” characters since they are unambiguous in each instance. Correcting these characters using the Levenshtein distance approach proves somewhat simpler when it involves swapping or deleting characters rather than inserting non-detected characters.

When following the correction instructions, if a character is to be exchanged for another character, only the character’s name needs to be modified. If a character that was detected initially is deleted during the correction, the label associated with it simply needs to be removed. When inserting a character, its position must be determined by referencing the uncorrected characters and considering the textual structure of the existing rolling mark to ensure accurate placement of the missing character.

However, it is important to handle cases like the one depicted in Figure 9 differently for all correction processes. In this scenario, the detection itself is accurate, but according to the correction instructions, the grounding cable would be incorrectly labeled as part of the month of manufacture.

6.2. Anomaly Detection

The recorded field of view captured by the camera provides additional valuable information beyond the rolling marks. It enables the detection of anomalies, such as missing irons, track holes, and other defects. By analyzing the captured images, it is possible to identify and address these irregularities, contributing to the overall maintenance and safety of the railway infrastructure (Figure 10).

7. Patents

In the course of this work, a patent (WO2022253660A1) was filed.