Skip to main content
SpringerLink
Log in

A self-adaptive heterogeneous multi-core architecture for embedded real-time video object tracking

  • Special Issue
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Sequential Monte Carlo (SMC) represents a principal statistical method for tracking objects in video sequences by on-line estimation of the state of a non-linear dynamic system. The performance of individual stages of the SMC algorithm is usually data-dependent, making the prediction of the performance of a real-time capable system difficult and often leading to grossly overestimated and inefficient system designs. Also, the considerable computational complexity is a major obstacle when implementing SMC methods on purely CPU-based resource constrained embedded systems. In contrast, heterogeneous multi-cores present a more suitable implementation platform. We use hybrid CPU/FPGA systems, as they can efficiently execute both the control-centric sequential as well as the data-parallel parts of an SMC application. However, even with hybrid CPU/FPGA platforms, determining the optimal HW/SW partitioning is challenging in general, and even impossible with a design time approach. Thus, we need self-adaptive architectures and system software layers that are able to react autonomously to varying workloads and changing input data while preserving real-time constraints and area efficiency. In this article, we present a video tracking application modeled on top of a framework for implementing SMC methods on CPU/FPGA-based systems such as modern platform FPGAs. Based on a multithreaded programming model, our framework allows for an easy design space exploration with respect to the HW/SW partitioning. Additionally, the application can adaptively switch between several partitionings during run-time to react to changing input data and performance requirements. Our system utilizes two variants of a add/remove self-adaptation technique for task partitioning inside this framework that achieve soft real-time behavior while trying to minimize the number of active cores. To evaluate its performance and area requirements, we demonstrate the application and the framework on a real-life video tracking case study and show that partial reconfiguration can be effectively and transparently used for realizing adaptive real-time HW/SW systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ali, U., Malik, M., Munawar, K.: FPGA/soft-processor based real-time object tracking system. 5th Southern Conference on Programmable Logic (2009)

  2. Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. Signal Process. 50(2), 174–188 (2002)

    Article  Google Scholar 

  3. Carvalho, E., Calazans, N., Moraes, F.: Heuristics for dynamic task mapping in NoC-based heterogeneous MPSoCs. International Workshop on Rapid System Prototyping (RSP) (2007)

  4. Cho, J.U., Jin, S.H., Pham, X.D., Kim, D., Jeon, J.W.: A real-time color feature tracking system using color histograms. International Conference on Control, Automation and Systems (2007)

  5. Curtis-Maury, M., Dzierwa, J., Antonopoulos, C.D., Nikolopoulos, D.S.: Online power-performance adaptation of multithreaded programs using hardware event-based prediction. International Conference on Supercomputing (2006)

  6. Doucet, A., de Freitas, N., Gordon, N.: Sequential Monte Carlo Methods in Practice. Springer, Berlin (2001)

  7. Gilbert, A.L., Giles, M.K., Flachs, G.M., Rogers, R.B., Hsun, U.Y.: A real-time video tracking system. IEEE Trans. Pattern. Anal. Mach. Intell. 2, 47–56 (1980)

  8. Happe, M., Lübbers, E., Platzner, M.: A multithreaded framework for sequential Monte Carlo methods on CPU/FPGA platforms. International Workshop on Applied Reconfigurable Computing (ARC) (2009a)

  9. Happe, M., Lübbers, E., Platzner, M.: An adaptive sequential Monte Carlo framework with runtime HW/SW partitioning. IEEE Int. Conf. Field Program. Technol. (FPT) (2009b)

  10. Hess, R.: Particle filter object tracking. http://web.engr.oregonstate.edu/hess (2006)

  11. Huang, C., Vahid, F.: Dynamic coprocessor management for FPGA-enhanced compute platforms. International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES) (2008)

  12. Kalman, E.R.: A new approach to linear filtering and prediction problems. Trans. ASME—J. Basic Eng. 82(Series D), 35–45 (1960)

    Google Scholar 

  13. Kobayashi, T., Nakagawa, K., Imae, J., Zhai, G.: Real Time object tracking on video image sequence using particle swarm optimization. International Conference on Control, Automation and Systems (ICCAS) (2007)

  14. Kumar, R., Farkas, K.I., Jouppi, N.P., Ranganathan, P., Tullsen, D.M.: Single-ISA heterogeneous multi-core architectures: the potential for processor power reduction. In: International Symposium on Microarchitecture (2003)

  15. Loza, A., Patricio, M., Garcia, J., Molina, J.: Advanced algorithms for real-time video tracking with multiple targets. 10th International Conference on Control, Automation, Robotics and Vision (ICARCV) (2008)

  16. Lübbers, E., Platzner, M.: ReconOS: multithreaded programming for reconfigurable computers. ACM Trans. Embed. Comput. Syst. 9(1), 1–33 (2009)

    Google Scholar 

  17. Nollet, V., Avasare, P., Eeckhaut, H., Verkest, D., Corporaal, H.: Run-time management of a MPSoC containing FPGA fabric tiles. Trans. Very Large Scale Integr. Syst. (2008)

  18. Porikli, F.: Achieving real-time object detection and tracking under extreme conditions. J. Real-time Image Proc. 1(1), 33–40 (2006)

    Google Scholar 

  19. Saha, S., Bambha, N.K., Bhattacharyya, S.S.: A parameterized design framework for hardware implementation of particle filters. IEEE International Conference on Acoustics, Speech and Signal Processing (2008)

  20. Sankaranarayanan, A.C., Chellappa, R., Srivastava, A.: Algorithmic and architectural design methodology for particle filters in hardware. International Conference on Computer Design (2005)

  21. Sigdel, K., Thompson, M., Pimentel, A.D., Galuzzi, C., Bertels, K.: System-level runtime mapping exploration of reconfigurable architectures. Proceedings of the International Symposium on Parallel & Distributed Processing (IPDPS) (2009)

  22. Sironi, F., Triverio, M., Hoffmann, H., Maggio, M., Santambrogio, M.: Self-aware adaptation in FPGA-based systems. International Conference on Field Programmable Logic and Applications (FPL) (2010)

  23. Smit, L.T., Smit, G.J.M., Hurink, J.L., Broersma, H., Paulusma, D., Wolkotte, P.T.: Run-time assignment of tasks to a heterogeneous processors. In: Embedded Systems Symposium (2004)

  24. Smit, L.T., Hurink, J.L., Smit, G.J.M.: Run-time mapping of applications to a heterogeneous SoC. Proceedings of the International Symposium on System-on-Chip (2005)

  25. Stitt, G., Lysecky, R., Vahid, F.: Dynamic hardware/software partitioning: a first approach. Proceedings of the Design Automation Conference (DAC) (2003)

  26. Visentini, I., Snidaro, L., Foresti, G.L.: Cascaded online boosting. J. Real-time Image Proc. 5(4), 245–257 (2010)

    Google Scholar 

  27. Woelk, F., Schiller, I., Koch, R.: An airborne bayesian color tracking system. In: IEEE Intelligent Vehicles Symposium (2005)

Download references

Acknowledgments

The research leading to these results has received funding from the European Union Seventh Framework Programme under grant agreement no. 257906. This work was supported by the International Graduate School of Dynamic Intelligent Systems and by the German Research Foundation under project number PL471/2-1.

Author information

Authors and Affiliations

  1. Computer Engineering Group, Warburger Str. 100, 33098, Paderborn, Germany

    Markus Happe & Marco Platzner

  2. EADS Innovation Works, Technical Capability Center 5, 81663, Munich, Germany

    Enno Lübbers

Authors
  1. Markus Happe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enno Lübbers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marco Platzner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Happe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Happe, M., Lübbers, E. & Platzner, M. A self-adaptive heterogeneous multi-core architecture for embedded real-time video object tracking. J Real-Time Image Proc 8, 95–110 (2013). https://doi.org/10.1007/s11554-011-0212-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-011-0212-y

Keywords

Search

Navigation