Tagnoo: Enabling Smart Room-Scale Environments with RFID-Augmented Plywood

2.1 Computational Materials

"Computational materials" is a term coined by Gregory Abowd, which refers to everyday materials that possess embedded computational capabilities or wireless sensors that seamlessly integrate into the material itself [29]. The concept of computational materials can be traced back to 1998 with the introduction of the Musical Jacket [72]. This innovative garment featured a fabric-based touch keypad, enabling users to interact with a computer using the jacket itself. Since then, extensive research efforts have been devoted to the advancement of interactive materials [48, 71, 74, 75, 94, 95]. A notable example relevant to this research is Capacitivo, developed by Wu, et al. [95]. Capacitivo features a fabric sensor that uses capacitive sensing to recognize the objects in contact with it. This broadens the range of applications for interactive materials, allowing for more intuitive and seamless user interactions.

Recent advancements in materials science and manufacturing have greatly accelerated progress in computational materials. These materials can now be produced at a low cost [35, 37, 96, 101], and even operate without the need for a battery [33, 34, 101], bringing practical applications within reach. For instance, the development of triboelectric nanogenerator (TENG) technology demonstrates the ability to convert mechanical energy to electrical energy [92, 93]. This breakthrough has paved the way for creating batteryless computational materials [33, 34]. One example of this progress is SATURN and MARS [34, 35], where researchers have successfully designed a batteryless paper microphone. This novel device utilizes TENG to sense audio sounds from users and employs backscatter to eliminate the need for a battery. Another development in this field is the integration of TENG into plywood, known as iWood [96]. By optimizing the structure of TENG, researchers have ensured that the sensor remains functional even during common woodworking operations. This integration enables the fabrication of everyday objects that are inherently embedded with vibration sensors, expanding the possibilities for incorporating computational capabilities into various household items.

Despite the significant progress made in the field of computational materials, most existing materials lack the capability to operate without wire connections for power, computation, or data communication. Moreover, most of them are incapable of sustaining their functionality in the event of damage during the fabrication process, such as cutting and screwing. In light of existing research, our work introduces a new method, which aims to create computational materials that effectively address these limitations. The proposed approach involves the integration of RFID tags into plywood, thereby imbuing the material with limited sensing, computing, and wireless connectivity capabilities, while still retaining the inherent characteristics of traditional plywood.

2.2 RF Sensing

Radio frequency (RF) signals have a rich history of utilization in the detection and tracking of objects, humans, and environmental changes [43, 84]. The underlying principle involves the transmission of a radio signal, followed by the observation of the reflected signal. This reflected signal contains information about the material composition, shape, size, orientation, and even movement patterns of nearby objects. By analyzing these reflected signals, a system can facilitate a wide range of applications, including the recognition of human activities [39, 61, 63, 67, 81, 83, 91, 105], the identification of gestures [32, 83, 90, 94], the monitoring of vital signs [30], the localization of indoor positions [65, 107], the identification of individuals [47, 55], and the recognition of objects [44, 59, 62, 79, 86, 102]. These advancements highlight the potential of RF sensing technology. Moreover, RF sensing offers distinctive advantages over alternative sensing approaches, as it can effectively identify objects that are obstructed by barriers and can even discern the materials comprising those objects. Building upon this foundation, our work aims to incorporate RF sensing into computational plywood, thereby enabling object recognition capabilities.

The RF-based technique that is directly relevant to our research is RFID. RFID is a specialized RF technology primarily used for the purpose of identifying and tracking objects or people using RFID tags. These tags are composed of a small chip and an antenna that can store and transmit data wirelessly to RFID readers. When an RFID reader emits an RF signal, it powers the tag, prompting it to respond with the information it holds. The applications of RFID technology are vast, with significant usage in supply chain management [87], asset tracking [46, 73], inventory control [80, 108], access control systems [77], and contactless payment methods [31]. Its versatility lies in its ability to operate at different radio frequencies, with the high-frequency range (13.54 MHz) and ultra-high frequency range (915 MHz) being the most popular for RFID systems. In HCI, both HF and UHF RFID have been utilized to track the location and orientation of objects [66, 85]. UHF RFID has demonstrated significant potential in various applications owing to its shorter wavelength, higher reading rates, and extended sensing range. Beyond object identification, UHF RFID tags possess the capability to enable functionalities such as activity recognition [30, 52, 68, 82], touch sensing [54, 76], shape awareness [57, 88], material identification [50, 89, 99, 106], person identification [47], and object imaging [89]. However, a significant challenge in deploying UHF RFID tags lies in the integration process, which demands substantial time, effort, and technical expertise from workers. Overcoming this obstacle is crucial to harness the full potential of UHF RFID technology. In response, our work demonstrates an approach that involves the integration of dense arrays of RFID tags into materials, specifically plywood. This approach empowers woodworkers to effortlessly and implicitly deploy an extensive network of RF tags across different items within an environment. By embedding RFID tags into plywood, the deployment process becomes simplified, enabling efficient and widespread adoption of UHF RFID technology in various contexts.

2.3 Sensor Deployment for Smart Environments

Smart environments are physical spaces that incorporate a range of computing and interface devices, working in tandem to enhance the overall quality of life, optimize resource utilization, and deliver personalized services to the individuals residing within these environments [41, 70]. Several key components are required to establish such smart environments, including connected sensors [58, 78], actuators [78], data processors [42], AI technologies [100], and software platforms [53]. However, deploying connected sensors is considered the foundational step among these components, as they play a critical role in collecting and providing data to the system, thereby driving its intelligence.

One widely adopted approach for deploying connected sensors in various environments involves directly attaching sensing devices to existing infrastructure in an ad-hoc manner [49, 51, 56, 61, 103, 104]. Extensive prior research has employed this approach by installing electromagnetic (EM) [40, 49, 61, 81, 105], vibration [51, 56, 97, 104], and light sensors [60, 64, 101] within the environment to detect and monitor user interaction and activities. For example, ElectriSense [49] installs sensors on power outlets, utilizing electromagnetic interference in powerlines to identify appliances. Similarly, Wall++ [105] utilizes painted antennas and sensors on walls to capture airborne EMI signals, enabling the detection and localization of appliances. Another notable work, StarLight [64], deploys sparse photodiodes on the floor to receive light signals and reconstruct human posture. Furthermore, efforts have been made to install vibration sensors or tags on floors and walls [104], as well as plumbing, gas, and HVAC infrastructure [103] to detect user activities. However, it is worth noting that the integration of sensors into existing spaces through current ad-hoc methods often results in devices that are visually conspicuous and do not seamlessly blend with the surroundings. While this lack of integration can be mitigated by concealing the sensors more effectively within the environment [105], this process typically requires expertise in electronics and engineering, thereby posing challenges for workers without a technical background.

In summary, our research stands out from the existing literature for its unique focus on the advancement of computational materials and their integration into practical use. Tagnoo is developed to withstand operations such as cutting, nailing, and screwing. This sets our work apart from many existing approaches as their sensors did not involve the same use of materials, causing them to break upon encountering such operations. Furthermore, unlike many other existing works, our approach does not rely on wired connections for power, computation, and data communication. This ensures that Tagnoo can seamlessly integrate into existing woodworking workflows and tools without requiring workers to acquire new skills unrelated to their domain, such as electronics or engineering.

4.1 Experiment 1: Tag Types

UHF RFID tags are available in various sizes and shapes, each tailored to meet specific application requirements. Our study focused on identifying a UHF RFID tag that could be seamlessly integrated into plywood, effectively operate in a dense configuration, and maintain a consistent reading regardless of its orientation. Since RFID tags may be positioned in various orientations inside furniture or floors, we needed to select a tag to accommodate these variations.

Tag pre-selection. Our selection process was guided by four criteria. First, the RFID tags should be easily cut or drilled to prevent obstruction during woodworking operations. Secondly, they need to be relatively small to allow for a decent sensing resolution in a 2D space when arranged in a grid layout. Thirdly, the reading range of the RFID tags should be sufficient to cover the size of a room. Lastly, the RFID tags should be reliably detected regardless of their orientation to the RFID antenna. We investigated 24 widely available RFID tags, including those with and without enclosures, and with varying sizes and reading ranges [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. We cut and drilled the tags with enclosures and compared their resistance to that of regular plywood, concluding that enclosures hinder woodworking operations. Consequently, tags with protective shells [1, 10, 12, 15, 22] were excluded. Additionally, tags with footprints exceeding 10 cm on any side were disregarded [6, 9, 17, 18, 19, 20, 21], as they would be too large to be arranged in a dense grid for accurate sensing. Further, tags with a reading range shorter than 5 meters were not considered [3, 4, 5, 11, 16], as they may not be reliably detected at a room-scale. The remaining tags were either circularly polarized or linearly polarized.

Orientation effect. To evaluate the reliability of the remaining tags at different orientations to the RFID antenna, we conducted an experiment using two specific tags: Impinj H47 [13] and Impinj E62 [25]. The Impinj H47 was chosen due to its widespread use as a circularly polarized tag. Additionally, the Impinj E62 was selected to assess the performance of a linearly polarized tag, given its similar size to the Impinj H47 tag. While these tags may not represent the most optimal options in terms of size or sensitivity, they were suitable for gaining insight into the performance differential between circularly and linearly polarized tags.

In our experimental setup, we carefully arranged each type of RFID tag under investigation into a 3 × 3 array configuration. The distance between the centers of adjacent tags was precisely set at 10 cm from one another. The purpose of our measurements was to evaluate the sensitivity of the tags in four different orientations: 0°, 30°, 60°, and 90°. For the circularly polarized tags, we excluded the 90° angle from testing to avoid duplicating results, as it is identical to the 0° angle. To ensure consistency, each reading phase was conducted for a duration of 3 minutes (approximately 6000 sample points per tag). Throughout the experiments, all the tags were positioned below the RFID antenna, and they were placed on top of a plywood surface. We retrieved RSSI from the RFID reader to measure the sensitivity of tags. To minimize the potential influence of tag location with respect to the RFID antenna, we calculated the average RSSI based on the data from all the tested tags.

Result. Figure 3 illustrates the results of the experiment. All of the circularly polarized tags demonstrated consistent readability, maintaining a stable RSSI across the four tested orientations. In contrast, the orientation of linearly polarized tags significantly impacted their RSSI. Specifically, during the 3-minute reading periods at 0° and 30° conditions, two and seven tags, respectively, could not be detected. Thus, we opted to incorporate circularly polarized tags into our prototype. Note that the close proximity between linearly polarized tags might affect their sensitivity, causing their RSSI to be lower than that of circularly polarized tags. Nevertheless, this fact does not alter our decision, because our decision was made based on the rationale that furniture in a smart environment should not be restricted to a specific orientation and should remain interactive even when rotated by the user.

Figure 3:

View Figure

4.2 Experiment 2: Tag Grid Density

This study aimed to examine the impact of the density of circularly polarized tag grids on tag readings. It is generally observed that a denser arrangement of tags results in a higher 2D sensing resolution, which is particularly advantageous for capturing subtle human activities and small objects in the environment. Furthermore, a denser arrangement is less susceptible to the loss of the sensing region within the plywood sensor caused by damage to the tags during woodworking operations like sawing, nailing, or screwing. However, the potential drawback of a denser tag arrangement is the possibility of interference between adjacent tags, leading to a decline in RSSI. Hence, our experiment aimed to determine the maximum density of RFID tags that can be used in our implementation without compromising RSSI. Like our previous experiment, we positioned the circularly polarized RFID tags in a 3 × 3 grid below the RFID antenna. The distance between the centers of adjacent tags was varied, ranging from 6 cm to 15 cm. To investigate the interference of the surrounding tags on the central tag, we calculated its RSSI. To avoid tag overlap at close ranges, distances below 5cm were excluded from this experiment.

Result. The results of this experiment are illustrated in Figure 4. Our findings indicate that the distance between tags has an effect on the RSSI. Specifically, we observed an improvement in RSSI as the distance between the centers of adjacent tags increased from 6 cm to 10 cm. However, once the distance exceeded 10 cm, we noticed a stabilization of RSSI, suggesting that the influence of adjacent tags on signal strength becomes minimal. This finding has implications for the design and deployment of our tag-based system. It suggests that maintaining a minimum distance of 10 cm between tags can optimize signal strength and minimize interference. Beyond this distance, the impact of adjacent tags on RSSI becomes negligible. However, it is important to note that there are trade-offs associated with increasing the distance between tags. One such trade-off is the effect on sensing resolution in a 2D space. If the target applications require a higher tag resolution, such as sensing a cup with a diameter of less than 10 cm, a balance needs to be struck. In such cases, it may be necessary to compromise on the optimal signal strength to ensure accurate object detection.

Figure 4:

View Figure

In certain scenarios, particularly those involving activities on the floor, the close arrangement of RFID tags may not always be necessary. Objects of larger size, such as feet in these scenarios, obstruct the propagation of RF signals over a wider area, allowing for a sparser arrangement of tags while still effectively detecting them. This approach offers the potential for reduced manufacturing costs. Thus, we conducted an additional experiment to investigate the minimum density at which tags can be arranged without significantly compromising the resolution of 2D sensing. The specific focus of our experiment was to determine the maximum horizontal distance at which a tag can detect an object (fist in our case), which measured approximately 10 cm in length. To carry out the experiment, a single tag was positioned beneath the RFID antenna, and the distance between the object and the tag was systematically varied from 10 cm to 30 cm, with an increment of 5 cm.

Result. The results of this experiment are illustrated in Figure 5. When the object is positioned within a distance of 15cm from the RFID tag, there is a notable decrease about 7dB in RSSI compared to the scenario without an object. This substantial change suggests that the presence of the object within this close proximity significantly affects the tag’s reception of RF signals. However, beyond a distance of 20 cm, this effect gradually diminishes and eventually becomes indistinguishable from the situation where no object is present.

Figure 5:

View Figure

Based on the results obtained from our experiment, we determined the maximum and minimum distances between adjacent tags. These findings hold implications for the development of Tagnoo in various item categories. For furniture items, such as desks, which commonly come into close contact with relatively smaller objects like lunchboxes, it is advisable to utilize a Tagnoo system with a dense tag arrangement. After careful consideration of available options, we have chosen a 10 cm distance in our implementation of plywood material with a high tag density. This decision was made to maximize signal strength, considering the marginal increase in 2D resolution compared to a relatively large loss in signal strength (e.g., approximately 2dB drop from 10 cm to 8 cm). On the other hand, when it comes to larger infrastructure components like floors, which typically encounter objects with larger footprints, such as feet, a more cost-effective approach can be adopted. In our implementation, we have positioned the Tagnoo tags at a distance of approximately 30 cm for plywood material with a low tag density (maximum 15 cm away from an object). This has led to a reduction in the fabrication cost of the system. Furthermore, we have adopted a 20 cm distance for the plywood materials with medium tag density.

4.3 Experiment 3: Depth of Embedded Tags

Plywood is a composite material that is formed by bonding multiple layers of wood veneers together. The face side of plywood typically shows a smoother and visually appealing surface. In our implementation, we have strategically embedded RFID tags within the veneer layers near the face side. It is thus important to consider the potential impact of the thickness of the top veneer layer on the signals emitted by the RFID tags. To investigate this, we conducted an experiment by embedding the 3 × 3 RFID tags within the veneer layers at varying depths from the plywood’s face surface. We examined the changes in signal strength across different thickness levels, including a 0.025 cm (0.01 inches) veneer and plywood sheets ranging from 0.32 cm to 2.54 cm thick.

Result. The results of this experiment are illustrated in Figure 6. It can be observed that the RSSI decreases as the thickness of the plywood layers increases. Specifically, when a veneer layer measuring 0.025 cm thick is placed over the RFID tag, there is a reduction of 1.7dB in the RSSI. Furthermore, as the thickness of the veneer layer increases beyond this point, the decrease in RSSI exceeds 3dB. For our implementation, we opted to cover the RFID tags with a veneer layer that is 0.025 cm thick to maximize signal strength.

Figure 6:

View Figure

4.4 Experiment 4: Gluing Strategies

During the fabrication process, a layer of veneer needs to be adhered to the RFID tags and plywood substrate. However, the sensitivity of these tags may potentially be compromised when covered with wood glue due to the moisture present in the glue. To understand the effects of wood glue on RFID tag performance, we conducted an experiment to evaluate its impact on the signal strength. Since the RFID tags are positioned near the face surface of the plywood, our investigation focused solely on examining the effects of wood glue applied on top of the 3 × 3 tags. Three adhesive strategies were tested: (1) Full Coverage, where both the RFID tags and the gaps between them were completely covered with glue, resulting in a uniform distribution of glue across the entire surface of the plywood panel’s interior (Figure 7 a). (2) Gap Filling, which involved filling only the gaps between the RFID tags and the edge of the plywood with glue, ensuring that the top surface of the tags remained glue-free (Figure 7 b). (3) Edge Application, where the glue was exclusively applied to the four edges of the plywood panel’s interior. This approach allowed the top surface of the tags and the majority of the surrounding gaps to remain free of glue (Figure 7 c). To cover the tags, a veneer layer with a thickness of 0.025 cm was used. For the purpose of this study, Gorilla Wood Glue was chosen due to its widespread usage in woodworking applications.

Result. The experimental results are depicted in Figure 7 d. Our findings indicate that applying wood glue directly onto the surface of RFID tags significantly attenuates the RSSI. However, when the glue is used solely to fill the gaps between the tags and the edges of the plywood, its impact on RSSI is minimal. As illustrated in Figure 7 d, there was a negligible difference in RSSI between Gap Filling and Edge Application. However, Gap Filling, due to its larger surface area for adhesion, potentially offers stronger bonding between the veneer and plywood substrate compared to Edge Application. Therefore, in our implementation, we apply wood glue exclusively to these gaps.

Figure 7:

View Figure

5.1 Experiment 5: Sawing

The goal of this experiment was to investigate the potential consequences of using a saw to cut plywood and its impact on RFID signals. We aimed to examine the impact of a damaged antenna on the signal strength of RFID tags. Specifically, a completely broken chip definitely leads to tag malfunction, but since our RFID tag has four antennas located at the corners, it is uncertain whether a tag, with a single cut leaving only two antennas, would still maintain its functionality. To investigate this, we included two common scenarios where tags undergo a straight-line saw cut: partially cut and fully cut. The partially cut condition aimed to replicate cuts that only damaged the antennas of the RFID tags (Figure 8 a), while the fully cut condition was designed to assess the impact when both the antenna and the matching circuit are damaged (Figure 8 b). We have intentionally excluded scenarios where the cut deviated from a straight line, as our discussion with woodworkers has indicated that the majority of common woodworking practices involve cutting panels in straight lines.

We conducted our experiment using a 3 × 3 array of RFID tags placed on plywood, with a distance of 10 cm between each tag. To simulate antenna damage, we deliberately cut a half portion and full portion of two antennas on the tags, as illustrated in Figure 8. To determine whether the polarity of the tag changes when only two antennas remain, we also tested their signal strength under a 90-degree rotation.

Result. The results of this study are presented in Figure 8, which includes a tag without any damage as a baseline. Encouragingly, the tested tags demonstrated resilience even when their antennas were partially damaged, although there was a noticeable decrease in RSSI. We did not observe a significant change in the tag’s polarity, as the RSSI values of the rotated tags were similar to those of tags without rotation. However, when the antennas were fully damaged, the tags ceased to function. We suspect that this is because the damage extended to the impedance matching circuit, located within 1.25 cm of the tag’s center. It is important to acknowledge that these results may vary depending on the specific characteristics of each tag. Nonetheless, this study highlights that our current implementation can withstand sawing during woodwork to a certain extent. From a sensing perspective, the most unfavorable outcome would be the loss of 10cm wide sensing areas along the edges of a cut board, which appears to be acceptable. The primary objective of this study was to investigate the damages caused by a saw. However, it is reasonable to infer that the findings can be extrapolated to damages caused by drills. Although drills may have different action characteristics, they share common features with saws in terms of the resulting damage to the tag, particularly in causing an open cut. Therefore, it is plausible to generalize the observed patterns in this study to damages caused by drills.

Figure 8:

View Figure

5.2 Experiment 6: Screwing

When inserting screws or nails, there is a potential for damage to occur to a tag. However, the nature of this damage may differ from that caused by cutting or drilling. This is due to the conductive property of screws, which allows for a capacitively coupled connection to the antenna. As a result, the screw may act as an extension of the antenna, enabling it to receive and emit RF signals. In order to understand the impact of screws on tag functionality, we conducted a study in which we inserted screws at various positions of an RFID tag affixed to a plywood surface. The purpose of this study was to investigate the effects of screw placement on tag performance. Figure 9 a-c illustrates the three specific locations we selected for screw insertion: (1) on the outer antenna, (2) on the inner antenna, and (3) directly on the chip. These positions were chosen to cover some of the most common scenarios for potential damage to RFID tags.

Result. Figure 9 d presents the collected RSSI data for the various tested positions, along with a baseline measurement taken without any damage. Our findings suggest that the tags were able to withstand being screwed in as long as the screw did not cause any damage to the chip. Interestingly, unlike cutting, screw damage occurring on the inner antenna, close to impedance match circuit, did not result in the tag ceasing to function. Instead, it only caused a slight decrease in the RSSI. This observation can be attributed to the conductive nature of the screw, which allows it to act as a functional component of the tag. It is worth noting that this damage-prone area only accounts for a mere 6 % of the total tag area. Upon careful observation, no significant fluctuations in the RSSI values were detected.

Figure 9:

View Figure

5.3 Experiment 7: Painting

One of the final and crucial stages in the manufacturing process of wooden items is the application of paint. However, there is a lack of certainty regarding the potential impact that a thin layer of paint commonly used on wooden items may have on the signal strength of embedded tags. This uncertainty is further complicated by the presence of moisture in certain types of paint, such as water-based paints. To address this knowledge gap, we conducted an experiment aimed at testing the effects of two different types of polyurethane wood finishes on the signals of embedded tags. Specifically, we tested a water-based polyurethane wood finish from Varathane and an oil-based polyurethane finish from Minwax. These two finishes were chosen based on their popularity and widespread use.

To conduct this study, we developed three identical copies of our prototype based on the findings from previous experiments. Each prototype consisted of a plywood substrate with a grid of 3 × 3 RFID tags affixed to it. Additionally, we applied a 1mm thick veneer layer using wood glue onto the tags. To understand the impact of different paints on the RFID performance, we applied different types of paint to the two prototypes. One prototype was painted using water-based paint, another with oil-based paint, and the final one was left unpainted to serve as a baseline for comparison. Following the provided instructions, the surface of the prototypes was brush-painted, ensuring that an even layer of paint covered the entire surface. After painting, we allowed the paint to air dry for a period of 24 hours before conducting the test to measure the RSSI.

Result. The results of the study are presented in Figure 10. The findings indicate that there was a marginal decline in the RSSI when measured on painted surfaces in comparison to the baseline. Nonetheless, it is worth noting that despite this decline, the signals could still be consistently received in both water-based and oil-based painted conditions. These findings suggest that the presence of a light layer of paint does not significantly impede signal transmission. Therefore, it is unlikely that the performance of the sensor will be significantly affected by the presence of paint.

Figure 10:

View Figure

7.1 Hardware

A typical sensing environment created using Tagnoo will have hundreds of tags. To scan these tags, we used an R420 UHF RFID reader in conjunction with two RFID antennas (Vulcan 262006). Each antenna was strategically positioned to cover an adjacent area, resulting in a combined coverage of approximately 320 × 250 square centimetre. The antennas were mounted on the ceiling at a height of 200 cm above the ground. To effectively scan the environment, we switched between the antennas to read the tags within their coverage areas. In our implementation, the system operated at a sampling rate of approximately 1Hz. This sampling rate is sufficient for detecting the presence of static objects or simple user activities, such as sitting or standing on the floor. Note that higher sampling rates can be achieved by higher-end RFID readers or antennas, such as phased array antennas.

7.2 Data Processing Pipeline

In the data processing pipeline, the RSSI values of all the tags present in the environment were sampled and processed. To eliminate any ambient noise and account for baseline RF propagation, a background profile was subtracted from the RSSI readings. This subtraction enabled a more precise representation of the interactions between the tags and objects or users. The background profile was established by taking the maximum of the RSSI readings from all the tags within a sliding window of 5 seconds. It was automatically updated when no objects or users were detected in the environment, indicating a period of inactivity, or when no tag reported a change in RSSI values exceeding a preset threshold (3 dB).

When an object or user interacts with a piece of furniture or the floor within the environment, our system detects their presence by analyzing the changes in RSSI values from the tags embedded in the furniture. This detection process involves summing up the changes in RSSI values and comparing them against a predetermined threshold (3 dB). If the summed change exceeds this threshold, the system identifies the presence of an object or user. Once the system detects this presence, it proceeds to monitor the standard deviation of the summed changes. The system waits until the standard deviation falls below a specific threshold (2 dB), within a 5-second time window. This indicates that the signal has stabilized and is ready for further processing. Next, the system smooths the RSSI values using a minimum filter because an object attenuates RF signals. We compute the relative changes in RSSI values for each tag, using the background profile as the reference for initial values. This process generates a series of RSSI data for each tag, arranged into a matrix based on the tag location within the sensing surface of furniture or floor. This data is then converted into a low-resolution image, which is then upscaled using a factor equal to the tag distance. For instance, the sensor data of a chair surface with a 3 × 3 tag configuration, arranged at a medium density with a 20cm gap, is upscaled to a 60 × 60 heatmap image. Similarly, the sensor data of a desk surface with a 12 × 6 tag configuration, arranged at a high density with a 10cm gap, is upscaled to a 120 × 60 heatmap image. Following this, the upscaled images from all the sensing surfaces in the environment are combined to form a single image, which can be used for training and evaluating our machine learning model.

7.3 Machine Learning

We employed a Convolutional Neural Network (CNN) classifier to tackle the classification of our sensor data, treating it as an image classification task. To avoid overfitting and optimize computational footprint and memory usage, we carefully designed our classifier with a lightweight architecture. The architecture of our model comprises four convolutional layers, with 8, 16, 32, and 64 filters, respectively, all utilizing a 3 × 3 kernel size and Rectified Linear Unit (ReLU) activation functions. These convolutional layers were further augmented by max-pooling layers, which employed 2 × 2 pooling windows to effectively reduce the spatial dimensions during the feature extraction process. The flattened features were then passed through a dense layer, consisting of 128 units, for subsequent classification.

We train over 100 epochs with a batch size of 16. Each epoch represents a complete pass through the entire training dataset, allowing the model to learn and adjust its weights. The selected batch size of 16 strikes a balance between computational load and frequency of weight updates, thereby facilitating smoother convergence. Throughout the training phase, the model undergoes forward passes, generating predictions, and subsequently comparing them against the actual labels using categorical cross-entropy as the designated loss function. The backpropagation process follows, adjusting the model’s weights to reduce error. The ’adam’ optimizer aids in this by determining the learning rate and other parameters. At most, our lightweight CNN classifier contains approximately 2 million parameters, resulting in a model size of around 16 MB. This places our classifier in a similar range, in terms of size and complexity, as traditional classifiers, such as the Random Forest classifier, with a substantial depth (equivalent to approximately 32 trees).

9.1 Objects and Activities

In our experiment, we tested a total of 18 objects and user activities, primarily selected from the scenario described in Section 3 (Figure 13). Specifically, we tested 8 events on the desk, including the presence of a 16-inch laptop, a user typing on the laptop, a chocolate bar, a craft hardboard, a book, a glass lunchbox, the glass lunchbox filled with rice, and a backpack. For the bookshelf, we conducted tests on 4 events, which involved placing a book on the top, middle, and bottom shelf, as well as having a user standing next to the shelf. On the chair, we tested 3 events, which included a user sitting on the chair, an empty backpack placed on the chair, and the same backpack with the 16-inch laptop inside. Lastly, on the floor, we tested 3 events, which involved a user sitting, standing, and lying on the floor. The selection of object and event locations was based on their natural and logical occurrence within the tested environment.

Figure 13:

View Figure

9.2 Participants

Ten right-handed participants were recruited for the user activities, including typing on a laptop, sitting on a chair, standing next to a shelf, and various positions on the floor (standing, lying, and sitting). The participants had an average age of 22 years, with five males and five females. A volunteer carried out data collection for the objects.

9.3 Data Collection

A volunteer was invited to collect the training data for the machine learning model. The volunteer was asked to place each of the tested objects inside the sensing area of each item in a random location, orientation, and order. No other instruction was given in terms of how the objects should be presented to the sensor.

For the data on user activities, participants were instructed to carry out each activity ten times without any restrictions on their body position or orientation within the sensor. For instance, when typing on the laptop, participants placed the laptop anywhere on the table and proceeded with typing. To simulate searching for books near the bookshelf, participants varied the trials by choosing different locations in front of the shelf. For activities performed on the floor, participants stood and sat at different locations with random body orientations. When lying on the floor, half of the trials had the participant’s head positioned on one end, while the other half had the head positioned at the opposite end. The raw data confirmed that objects and activities occurred across a wide variety of locations and orientations, including those in close proximity to the center, as well as those took place near the edges or corners of the sensors.

As described in the Data Processing Pipeline section, the sensor data obtained from the entire environment, including the table, chair, bookshelf, and floor, was combined into a single heatmap image for model training and testing.

9.4 Results

In this section, we present the results of our study on the recognition accuracy of Tagnoo. Particularly, we present the system’s performance by comparing the use of a single general model for the entire environment with the use of individual models for each item in isolation.

Figure 14:

View Figure

9.4.2 Item-specific model accuracy.

In addition to assessing the overall accuracy of our general model, we measured sensing accuracy for each individual piece of furniture and the floor. To do this, we performed a two-fold cross-validation for each model trained using the specific data collected for each item while excluding data from the remaining environment. The results also revealed a high level of accuracy for each item, with the table achieving 95.3% accuracy (SD = 4), the bookshelf achieving 91.2% accuracy (SD = 6.8), the chair achieving 94.2% accuracy (SD = 9.0), and the floor achieving 91.6% accuracy (SD = 4.7). The corresponding confusion matrix can be found in Figure 15.

Figure 15:

View Figure

The results of the study indicate instances where the sensing accuracy was compromised when a complete picture of sensor data from the surrounding environment was not available. Specifically, the identification of joint events occurring across different pieces of furniture was hindered. For instance, when the data from the chair is excluded from the model, there is a misclassification of typing on the laptop, often confusing it with the laptop being placed solely on the table. This misclassification can be attributed to the absence of data regarding someone sitting in the chair, causing the system to inaccurately attribute the activity solely to the table. To further investigate the influence of missing the surrounding environment, we intentionally incorporated the data collected on the chair with participants standing between the chair and the bookshelf. Unlike in the general model, the data collected on the bookshelf was excluded. The analysis revealed that the model frequently confuses situations where only a bag is present on the chair with situations where a person is standing between the chair and the shelf. This finding suggests that when developing a model for individual smart items, a more comprehensive set of test scenarios should be considered to minimize false positives. Interestingly, we did not observe any significant increase or decrease in average accuracy for the bookshelf and floor. This can be attributed to the fact that in the tested environment, the events that took place on these items were more dedicated to the items themselves.

10.1 Detecting and Locating Multiple Objects or Users

Our current implementation relies on a simple machine learning algorithm to detect the presence of an object and user by analyzing the RSSI values of the tag grid. While this approach works well for simple scenes, it may not be suitable for more complex scenarios involving the simultaneous locating and tracking of multiple objects or users, particularly in the presence of multi-path signals. To tackle this challenge, utilizing a deep learning model might offer a more robust solution [59, 98]. However, incorporating deep learning models into our system is not without its difficulties. One major concern is the potential interference between objects, which can hinder the accurate sensing of material information. For instance, when two objects are in close proximity, their RSSI patterns may overlap, leading to difficulties in distinguishing between them. This overlap poses a significant challenge that needs to be addressed. One possible approach to address this challenge is by employing a super resolution network model [38, 69]. This approach has the potential to generate higher-resolution data by uncovering hidden information within the low-resolution data. By enhancing the resolution of the data, it may become easier to distinguish between overlapping RSSI patterns and accurately identify individual objects or users. However, it is important to note that further exploration and validation are required to determine the effectiveness of this approach.

10.2 Latency

Our current research primarily centers on material-related concerns. As a result, certain technical improvements have been omitted from the scope of this study. A notable technical limitation is the sampling rate limitation of our current system, which poses a challenge to work effectively in an environment with a higher number of RFID tags or to accurately recognize fine-grained user activities, such as writing and erasing on a desk. To address this limitation, we propose two potential solutions: adaptive sampling and beamforming. Adaptive sampling is a technique that strategically allocates computational resources to specific areas of interest while minimizing focus on peripheral regions. In our context, this means that we can prioritize the sampling of RFID tags that are closer to the occurrence of an event of interest. This approach can be achieved during the tag singulation process, where only the tag corresponding to the serial string provided by the system is activated. By selectively sampling tags in proximity to the event, we can improve the sampling rate of the system. Another promising solution is beamforming, which involves directing RF signals in specific directions. This reduces conflict resolution time, as fewer tags are present in the targeted scan areas. Additionally, beamforming enables the coordination of multiple RFID readers to scan various regions simultaneously, thereby enhancing the overall sampling rate. By combining adaptive sampling and beamforming techniques, we can significantly increase the RFID reading rate and potentially achieve fine-grained activity recognition in a larger-scale environment.

10.3 Sensing Beyond a Fixed Environment

Our current sensing approach operates effectively in a fixed environment where the furniture’s location remains constant. However, in real-world scenarios, it is common for changes to be made regarding the arrangement and layout of household items within the environment. Additionally, the location or orientation of the household items may also be subject to change, which can affect the sensor readings and subsequently impact recognition accuracy. To address this issue, it is essential to develop a generalizable model that takes into account the arrangement layouts of items and the locations of the RFID antenna. This information can be manually provided by workers during the construction of the environment, or it can be automatically estimated using RFID readers equipped with localization techniques [45, 65, 107], such as Impinj xArray Gateway RFID Reader [28]. By incorporating this information, the system could potentially accommodate the changes within the smart environment. Nevertheless, this approach necessitates careful investigation in the future to validate its effectiveness.

10.4 Capabilities Beyond RF Sensing

The focus of our research revolves around computational materials that possess sensing, computing, and communication capabilities. We developed Tagnoo as an example of such a material due to its batteryless nature, cost-effectiveness, and resilience against fabrication operations. Tagnoo’s inherent computing and communication abilities make it a promising candidate for easy integration with additional functionalities beyond RF sensing. One possible augmentation for RFID tags is capacitive sensing. Recent studies have explored the potential of enhancing RFID tags with capacitive sensing capabilities [36, 88]. This additional feature expands the range of applications that can be addressed by Tagnoo. Moreover, the memory capacity of RFID tags can be leveraged to enable novel applications. For instance, objects created using Tagnoo can store meta-information within the memory of the RFID tags. This information can then be accessed and manipulated through augmented reality AR interfaces. Such an integration would enable users to interact with the physical environment in a more immersive and dynamic manner, bridging the gap between the physical and virtual worlds.

10.5 Comparison with RF Sensing Using mmWave Radar

It is worth noting that mmWave radar sensors offer similar sensing capabilities but through the use of electromagnetic waves for capturing the three-dimensional geometry of an environment. These sensors can be likened to "cameras" that operate using high-frequency waves. One distinct advantage of mmWave radar is its ability to function effectively in various environmental conditions, including low light or obstructed environments such as walls or fog. However, mmWave radar tends to exhibit lower sensing resolution when the sensing range is long [38]. In contrast to mmWave radar, the approach adopted by Tagnoo leverages the density and strategic placement of RFID tags in order to optimize sensing resolution. While our current implementation may not offer an extended sensing range, this limitation can be overcome by incorporating higher-end RFID readers and antennas [26]. Moreover, mmWave radar systems often prove to be expensive and power-intensive, particularly for long-range applications, which contradicts the goals of computational materials. In contrast, our system provides a more energy-efficient and cost-effective solution. At present, we are actively exploring methods to extend the reading range of Tagnoo without compromising the sensing resolution. By doing so, we aim to enhance the overall effectiveness and functionality of our system.

10.6 Recycle of Tagnoo

The integration of RFID tags in plywood poses new challenges in the recycling process. If these tags are not disassembled properly, they could contaminate recycled wood streams or emit harmful substances during incineration. To address these issues, our future research will focus on a two-step solution. Firstly, we plan to refine the design of Tagnoo to incorporate an easy-to-dismantle mechanism. Our objective is to develop a design that guides users to easily access and remove the RFID tags before initiating the recycling process. Secondly, we aim to develop spectral sensing and analysis tools that can be deployed at recycling centers. These tools will enable the seamless identification and separation of Tagnoo products from regular plywood. This way, we can accurately distinguish Tagnoo products, based on their unique characteristics and ensure that different components of Tagnoo receive the specialized recycling treatment it require.

10.7 Upcycling

Tagnoo offers a unique opportunity for the upcycling of furniture and household items, transforming them into new creations that serve different purposes. Similar to existing furniture, Tagnoo can be repurposed into various items, granting them a second life. However, what distinguishes Tagnoo is its digital properties, which can be inherited by these new items. This inheritance enables the upcycled items to become interactive and seamlessly integrate into the smart ecosystem.

10.8 Privacy Control

Users may have a desire to disable the sensing functionality of Tagnoo in certain situations. Although the entire smart environment can be deactivated by turning off the RF antennas and readers, selectively disabling a specific item composed of Tagnoo furniture presents a challenge at present. To tackle this issue, one potential solution is to design a user-friendly toolkit that enables users to easily shield the tags embedded within the plywood, by utilizing a metallic cover. Another potential solution involves the creation of methods to enable the RFID reader to bypass the group of tags within a desired item.

10.9 End-User Tools

End-user tools play a crucial role in the democratization of computational material. In our future research, we plan to develop end-user tools that will facilitate the seamless configuration of tag positions and distributions within diverse material designs. Additionally, we aim to design tools that will ease the machine learning pipeline, guiding users through the process of data collection, model training, and implementation in real-world scenarios. With these tools, our objective is to empower end users, irrespective of their technical background, with the capability to customize the functionality of their Tagnoo-enhanced objects to meet their specific needs and preferences.