Service Platform for Real-World Internet Integration in Mobile Applications - Dannewitz

Service Platform for Real-World / Internet Integration in Mobile Applications
Christian Dannewitz1, Holger Karl1, Daniel Warneke2, University of Paderborn1, TU Berlin2, Germany

Real-world activities of users could tremendously be supported by Internet services if those services were based on a seamless real-world integration that does not disrupt the users' work flow - especially in mobile situations. But such Internet services are currently very difficult to develop on a large scale because conceptual network support for such applications is missing. In this paper, we focus on providing network support for Internet services that implement real-world-integrated use cases. Use cases and the resulting developer requirements are analyzed. Based on this analysis, we propose a new paradigm, the Augmented Internet, that makes the real-world integration an integral part of the network. This builds a conceptual, bidirectional connection between the Internet and the real world, directly supported by the semantics of the communication network. To realize this new paradigm, we have designed a new networking architecture based on an information-centric network. This new architecture facilitates and accelerates the development of a completely new family of real-world-integrated Internet services.



The Internet plays a constantly increasing role in our everyday life and this trend will further accelerate in the coming years. Yet, interacting with the Internet is still a detached, computer-centered task, poorly integrated with other activities in the real world. Hence, there is an increasing need for real-world activities to be better integrated with Internet activities. Currently, the Internet coexists like a virtual world in parallel to our real world. The poor integration between Internet- and real-world activities is especially problematic in situations where the user is mobile. Here, current use cases are mainly restricted to mobile surfing. The mobile user has to stop his current work flow in the real world, start a web browser, and search for the desired information on the Internet. The recent, strong trend towards GeoWeb applications like Google Earth, Microsoft Virtual Earth, and NASA Earth Wind [1] is a good example that illustrates the strong user request for a closer integration of Internet information and the real world. Nevertheless, GeoWeb applications are only the first step, they still represent mainly computer-centered services that integrate Internet-based information only with a simulation of the real world. In comparison to current Web applications, true realworld-integrated Internet applications put much higher requirements on an intuitive and direct access to the online information and services. These applications should not force their users to concentrate on them, but support their users in pursuing their activities in the real world. In this paper we argue that the challenges that arise from these requirements can be tackled best on a network level. We explain why the current Internet architecture does not support the development those appli-

cations sufficiently and present a new informationcentric approach. This approach provides a conceptual integration of real world and Internet on the network architecture level and is called Augmented Internet. An Augmented Internet could facilitate the application development for many new use cases and provide a common infrastructure. E.g., it could enable the interaction with physical entities (like objects, people, and places) via entity-related Internet services directly from within the real world. This would enable a completely new real-world interface to access physicalentity-related Internet services. Such services should be accessible without requiring a conventional Internet search. E.g., instead of surfing the web to get architectural and historical information about a certain building during sightseeing, a tourist can simply “click on”1 the building on spot with his cellphone to obtain all relevant information. The information could then be presented in any desired way, e.g., via the cellphone display, via audio headsets, or via augmented reality glasses in case of images representing the building the way it looked 100 years ago. Besides requesting information about entities, the user could also execute entity-related operations. For example, a user could “click on” a library book, thereby executing the online renewal service to renew the book. Furthermore, a selection menu could be displayed to provide additional options like accessing the bibtex information of the book or storing private notes in a way that they are directly bound to the book instead of storing the notes in some separated text document that is difficult to find later on. The interaction with physical entities from within the Internet, e.g., to control intelligent web-enabled ap“Clicking on” an entity refers to identifying an entity and can technically be realized by, e.g., using RFID tags, bar code readers, or GPS

pliances, could also be simplified and unified by an Augmented Internet, making the Augmented Internet concept bidirectional. Such a concept could be based on virtual representations that represent physical entities on the Internet, serving as proxies where entityrelated services are cumulated. E.g., the virtual representation of a VCR can serve as proxy to remotely program the VCR from within the Internet. After discussing the relation to other work in Section 2, we extract the common user requirements from use cases of real-world-integrated applications and analyze the resulting application developer requirements (Section 3). Based on this analysis, we argue that a general new networking architecture is needed to support application developers in building new services that truly integrate with real-world activities. In Section 4, we present the AugNet architecture that is a specific implementation of the Augmented Internet concept. We discuss the core concepts and related design decisions, and we illustrate the conceptual integration into the network architecture.


User and Application Developer Requirements

We have divided the user requirements derived from examined use cases into three thematic groups. The following three subsections discuss those user requirements and describe resulting challenges for application developers.


Virtual Representations


Related Work

The need for real-world-integrated applications has been demonstrated by several projects. In an early project, Wellner [2] has proposed to couple information with physical objects. This concept has since been implemented successfully by multiple projects [3]–[9]. Most projects are based on custom-tailored solutions that could greatly benefit from a common infrastructure. The need for such a common infrastructure has been addressed by projects like Cooltown [10] and by Langheinrich et al. [11]. Those infrastructure projects are built on top of the current Internet architecture. Hence, they are based on a nodecentric communication paradigm that differs fundamentally from the information-centric paradigm of real-world-integrated applications. To support truly seamless integration, the infrastructure required by those applications becomes an integral part of the network in our architecture. No other architecture that we are aware of provides support for real-worldintegrated applications as an integral part of the network architecture in general and based on information-centric networks in particular. The information-centric network concept in general is closely related to the dissemination network concept as described by Jacobson [12] and to the DONA project [13]. A related concept of integrating content delivery mechanisms into the network is investigated by Plagemann et al. [14]. All those projects do not consider making real-world integration an integral part of the network. In our work, we evaluate and modify those concepts to build a new networking architecture based on the combination of the information-centric network paradigm and the Augmented Internet paradigm.

To enable users to access information and services related to a physical entity in an intuitive manner, many use cases require a single point of reference, also highlighted by previous projects [10]. We call the concept of a single point of reference that represents physical entities on the Internet a virtual representation. A virtual representation contains all information and services that are related to a certain entity like a book. When “clicking on” the book, the system presents the user a list of available related services. This list has to be limited to a short number of relevant hits, because sifting through numerous irrelevant hits as produced by conventional full-text search engines severely limits the smooth integration with real-world activities. Hence, a list of related information and services that is based on an unambiguous binding between the information and the entity itself is required by the application developer. To further reduce and customize the selection of displayed information, users should be able to create personalized virtual representations for entities, containing a customized selection of any kind of public information and potentially additional private data. For example, a user might want to add an Internet print service and some personal notes to his private book representation. Therefore, modifiable and extensible representations have to be supported. Moreover, the concept has to support multiple coexisting virtual representations for the same physical entity as different users might create personalized representations for the same entity. We call this concept multiplicity. For a deep integration with real-world activities, the necessity for user input must be limited as much as possible. Especially in the mobile domain, applications calling for much input can significantly disrupt the user from his current work flow. In order to specify a request with minimal user input, the Augmented Internet concept should support applications to enable preprocessing of entity-related information based on metadata. This metadata can be comprised of context information concerning the user’s current surrounding or device, his preferences, or entity relationship information like one-to-one/many relations, inheritance relations, and static/dynamic grouping. Grouping, for example, could enable a user to renew all library books he has on loan in a single step. In an ideal case, the application could reason the user intent solely

from the metadata and perform a request without further user input. An important question is how public virtual representations are generated on a large scale. A centralized content generation process is not viable and strongly undesirable for a world wide Augmented Internet from our point of view. Every Augmented Internet user should be able to generate virtual representations in a community-based approach. Community-based content generation has proved to be a viable approach to generate high quality content, as evidenced, e.g., by a Nature article [15] that concludes that the quality of Wikipedia articles goes “head to head” with the quality of online Encyclopedia Britannica articles.


Network Requirements for Seamless Integration with Real-World Activities


Bidirectional Real-World / Internet Binding

A main goal of the Augmented Internet concept is to support users during their real-world activities and provide direct access to online resources related to physical entities. Conventional search engines represent a detour to these resources, unnecessarily demanding the users’ attention. We try to eliminate these detours and let the user access the desired data while directly interacting with physical entities (e.g., by “clicking on” or “looking at” the entity). As a result, entities have to be identifiable based on their attributes in the real world. Unfortunately, there is no common way to identify different types of entities. Books can, e.g., be identified by “clicking on” them, implemented via a bar code or RFID, places can be identified via GPS, and buildings can be identified while “looking at” the building, implemented via image recognition or by using a combination of GPS and an electronic compass. Therefore, the Augmented Internet concept has to support multiple identification technologies, and a single entity could potentially be identified by different technologies or a combination of technologies. Furthermore, to handle the binding of virtual representations and physical entities, the concept requires a binding mechanism. To enable an unambiguous binding, physical entities require a separate, unique identifier. Using physical attributes as identifiers is not sufficient as technologies like GPS and image recognition are ambiguous, i.e., they are error prone and multiple entities can match the same attribute values, which would result in ambiguous identifiers. The Augmented Internet represents a conceptual, bidirectional connection between the real world and the Internet. Hence, besides accessing Internet-based information directly from within the real world, realworld entities should also be accessible from within the Internet, e.g., like programming a VCR remotely via the Internet. To facilitate such patterns, all components of an Augmented Internet implementation have to support bidirectional interactions.

Users are primarily interested in accessing information related to real-world entities, not in accessing nodes that host information, which represents the notion of the current Internet. This also applies to classical web browsing to some extent. In the Augmented Internet concept, this becomes even much more important as we consider the ability to directly and persistently address physical entities and related information objects as an important building block to minimize the distraction from real-world activities. Consequently, we advocate that the Augmented Internet should be based on a location transparent, information-centric network paradigm. This underlying network has to be extremely scalable and robust as an implementation of the Augmented Internet concept has to plan for a large scale system with millions of users and billions of virtual representations. To achieve the perception of a seamless integration of real-world activities with Augmented Internet applications, the network has to be very dependable and should even be able to handle interruptions in network connectivity in a failure transparent way to minimize delays during real-world tasks. Minimizing delays is especially important for mobile users as they may be highly involved in other tasks while accessing the network. As the Augmented Internet has a strong focus on mobile users, it is particularly important that the network inherently supports mobility transparency to support mobile users and mobile physical entities. To seamlessly present information to the mobile user, unobtrusive interfaces are required. Thus, several unobtrusive output interfaces should be supported by the Augmented Internet concept, including audio output and visual augmented reality technologies. Especially augmented reality poses strong requirements concerning low latency and high performance data access. To improve the performance in case of mobility and concurrent access, advanced information distribution mechanisms have to be supported by the network. This includes the performance transparent reconfiguration of the system.


AugNet - Implementing the Augmented Internet concept

In this section, we discuss implementation approaches for the Augmented Internet concept. In Subsection 4.1, we illustrate why the current Internet architecture is not well suited to serve as a platform for realworld-integrated applications. In Subsection 4.2, we present a new networking architecture, the AugNet, that solves the deficiencies of the current Internet architecture regarding the presented Augmented Internet

requirements. We conclude with an overview of security and privacy issues in Subsection 4.3.


Shortcomings of the Current Internet Architecture

The requirements of Section 3 can be summarized by three central concepts: a virtual representation of physical entities, a bidirectional, unambiguous binding between physical entities and their representations on the Internet, an information-centric network to store and access virtual representations.

Real-world-integrated applications can only be developed in a limited manner based on the current Internet architecture as today’s Internet offers only limited support for those concepts: (1) A sound concept to represent physical entities on the Internet does not exist. (2) The only currently available binding between physical entities and related information is based on searching information via full-text search engines. While full-text search engines are powerful tools to find certain types of information, they are not good at producing a small number of exact entity-related hits and they do not offer an efficient way to search for entity-related information by physical entity attributes. Many attributes that identify the entity (e.g., location, RFID tag) cannot currently be used to search for related information because they are either not part of the indexed data at all, or imprecise and error-prone and, hence, do not match a full-text search. (3) An information-centric network is currently missing. An information-centric network focuses on the dependable, transparent access to application information objects. This approach is fundamentally different from the current Internet that focuses on nodes and reliable byte streams between those nodes. P2P networks represent a first step towards an informationcentric concept. For example, they can partly provide location transparency but miss other important characteristics of information-centric networks like advanced, meta-information-based search mechanisms (e.g., via physical entity attributes) and sophisticated information distribution mechanisms.


AugNet Architecture

The AugNet is an implementation of the Augmented Internet concept. It is based on a distributed, information-centric network approach and consists of three

core conceptual components. Those components implement a new way to structure, store, and access entity-related information that differs radically from the way currently used in the Internet: Virtual Entities implement the virtual representation concept to structure physical-entity-related information. The Name Resolution System implements the bidirectional realworld / Internet binding concept and facilitates the physical-attribute-based information access. The Virtual Entity Repository is storing and providing transparent access to Virtual Entities. In the following paragraphs, we will first discuss each component and will then discuss the integration of those components with the information-centric network concept. Virtual Entities cumulate physical-entity-related information and services. This cumulation can be implemented in two different ways: either by tagging entity-related information spread all over the Web with an entity-specific identifier, or by providing a dedicated data structure for each physical entity that cumulates all entity-related information and services, e.g., via linking existing information. We have chosen to implement Virtual Entities via dedicated, flexible data structures for the following reasons. First, tagging is only possible with Web-based data, whereas it is not suitable for linking general Internet-based services to a physical entity. Unlike tagging, a flexible data structure can describe and, hence, support any kind of Internet-based service API. Furthermore, the multiplicity concept would be very difficult to implement based on tagging because every entity-related piece of data would have to be tagged with additional information, one tag for each personalized representation that it is part of. This results in many changes all over the network for each new or modified personalized representation. In contrast, such changes can easily be managed in a single place when using Virtual Entity data structures in combination with a separate binding mechanism as described below. Last, to support application developers with reasoning to reduce the need for user input, dedicated data structures can easily store metadata like an entity history and entity relations. Storing metadata in a distributed, taggingbased system would again be difficult because the metadata would be distributed all over the Web, frequently requiring extra searches for this metadata. The Name Resolution System manages information about physical entities and their attributes like GPS coordinates and RFID numbers, as well as binding information about which Virtual Entity(s) represent(s) which physical entity. The binding can also be implemented in two different ways: the first is again based on tagging while the second is based on a lookup service. To implement the binding via tagging and make Web information searchable via physical attributes of the related entities, Semantic Web technologies like RDF and OWL could be used to tag pieces of information on the Web with those attributes. But dynamically changing attributes like GPS

coordinates would result in tremendous permanent changes for every piece of information, making this approach inviable. In contrast, such changes could be managed in a single place when using a lookup service. Hence, a lookup mechanism was used to design the Name Resolution System, comparable to the Domain Name System but more sophisticated. The Name Resolution System can perform “fuzzy” searches (e.g., “find entities that are close to X.Y.Z”), can support multiple identification technologies, and can combine different technologies for searching. The Virtual Entity Repository builds the foundation of the AugNet by storing Virtual Entities and managing access to Virtual Entities via the Name Resolution System. To facilitate those responsibilities, an underlying network is needed that fulfills the requirements as described in Subsection 3.3, namely transparency, dependability, low latency, and high performance. Those requirements can nicely be fulfilled by the information-centric network concept. The following three concepts are essential for this: first, an identifier/locator split to support mobility- and location transparency, second, replication of application information objects, and third, advanced information distribution mechanisms based on object meta information. The combination of those three concepts facilitates dependable, failure transparent, low latency, high performance data distribution because information objects can be obtained from any available source. Local sources can be used in case of network connectivity interruptions, near-by sources to reduce latency, and parallel access to multiple copies can improve the network performance. As a result, an Augmented Internet based on an information-centric-network has several inherent advantages over an implementation that builds on the current Internet. In summary, it provides native support for location transparent information access, working in both directions based on any meta information. It offers inherently dependable, failure transparent access to virtual entities, and it can handle mobility. Because of those advantages, we design the core components of the AugNet architecture by using and extending information-centric network concepts. Information-centric networks represent data in application information objects. A special kind of selfcontained information objects is used in the AugNet architecture to represent Virtual Entities, coexisting with other types of information objects. The concept of information objects has to be extended by the AugNet architecture to implement the requirements of the Augmented Internet concept, especially multiplicity and the advanced metadata concept to represent inheritance, grouping, etc. Common information objects differ from Virtual Entities as Virtual Entities are retrievable via real-worldrelated physical attributes, which requires an advanced mapping and search mechanism. This ad-

Mobile Client
Augmented Internet Application

Information-centric Network

Phy_Attributes[] V_EntityNames[] V_EntityName VirtualEntity

Entity Identification Entity Identification


Name Name Resolution Resolution System System Virtual Virtual Entity Entity Repository Repository Servicespecific Server
(e.g., Web server)


Entity Selection Entity Selection


Service Selection Service Selection

ServiceRequest ServiceResponse

Service Execution Service Execution


Fig. 1

AugNet architecture

vanced mechanism – provided by the Name Resolution System – can also benefit other application information objects. Therefore, the Name Resolution System is integrated into the information-centric network by the AugNet architecture, building the foundation to search information objects based on any kind of meta information. Figure 1 shows the core components of the AugNet architecture and illustrates the communication flow between the AugNet components and a mobile client running an exemplary application that provides information by “clicking on” physical entities. When the user “clicks on” a physical entity, the physical entity attributes are measured and sent to the Name Resolution System, which returns a list of matching Virtual Entity names. A Virtual Entity is selected from the list and the Virtual Entity Repository returns all or a part this Virtual Entity. At last, an entity-related service is selected, e.g., to get information from a certain Web server, and executed by a service-specific server. This server is not part of the AugNet architecture and could be, e.g., an application server, Web server, etc.


Security and Privacy

In a system that integrates the Internet with the real world, privacy, trust, and control of personal data might be even more important than in the current Internet. Privacy mainly involves two aspects in the Augmented Internet. First, it has to be ensured that the user's privacy remains unaffected while he is using the system. This can, among others, be realized via anonymous read access to public virtual entities. Second, access to private data has to be restricted by the system, which can be implemented via a strict access control to and encryption of private virtual entities. Besides privacy, trust is an important aspect of the Augmented Internet, which can be implemented efficiently via self-certifying virtual entities, a concept adapted from the information-centric network concept.

Unlike the current Internet, where users have no influence on data once it is published, our system ensures that a user remains in much better control of his personal data. By aggregating all data that is related to a physical entity into a single Virtual Entity, our system allows much better control over data. It is a lot easier to find all information that is published about an entity, as well as to globally delete data. This can significantly reduce the effect of maliciously published compromising or otherwise harmful data about people or related entities. Originators of such data can be identified by controlling write access to virtual entities in an accountable manner and malicious entries can be deleted more easily in the Augmented Internet.



The tremendous potential of real-world-integrated applications can not be realized sufficiently based on the current Internet architecture. Hence, a new architectural paradigm is needed. The Augmented Internet paradigm supports real-world-integrated applications as an integral part of the network and provides advantages that can hardly be achieved at the middleware/application layer. Those advantages are demonstrated by the AugNet architecture that can provide excellent support for the information-centric nature of Augmented Internet applications as it is based on an information-centric network. We believe that the Augmented Internet builds the foundation for the next important evolutionary step towards a truly ubiquitous Internet.



[1] National Aeronautics and Space Administration. World Wind. [2] P. Wellner. Interacting with Paper on the DigitalDesk. Communications of the ACM, 36(7):87– 96, July 1993. [3] H. Ishii and B. Ullmer. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In CHI ’97: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 234–241, New York, NY, USA, 1997. ACM Press. [4] S. Pradhan, C. Brignone, J.-H. Cui, A. McReynolds, and M. T. Smith. Websigns: Hyperlinking Physical Locations to the Web. Computer, 34(8):42–48, 2001. [5] R. Want, K. P. Fishkin, A. Gujar, and B. L. Harrison. Bridging Physical and Virtual Worlds with Electronic Tags. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 370–377, New York, NY, USA, 1999. ACM Press.

[6] R. Barrett and P. P. Maglio. Informative Things: How to Attach Information to the Real World. In UIST ’98: Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology, pages 81–88, New York, NY, USA, 1998. ACM Press. [7] J. Rekimoto and K. Nagao. The World through the Computer: Computer Augmented Interaction with Real World Environments. In Proceedings of the 8th Annual ACM Symposium on User Interface and Software Technology, pages 29–36, USA, 1995. ACM Press. [8] P. Ljungstrand, J. Redstr?m, and L. E. Holmquist. WebStickers: Using Physical Tokens to Access, Manage and Share Bookmarks to the Web. In Proceedings of DARE 2000 on Designing Augmented Reality Environments, pages 23–31, NY, USA, 2000. ACM Press. [9] P. Debaty, P. Goddi, and A. Vorbau. Integrating the Physical World with the Web to Enable Context-Enhanced Mobile Services. Mobile Networks and Applications, 10(4):385–394, August 2005. [10] T. Kindberg, J. Barton, J. Morgan, G. Becker, D. Caswell, P. Debaty, G. Gopal, M. Frid, V. Krishnan, H. Morris, J. Schettino, B. Serra, and M. Spasojevic. People, Places, Things: Web Presence for the Real World. Mobile Networks and Applications, 7(5):365–376, October 2002. [11] M. Langheinrich, F. Mattern, K. R?mer, and H. Vogt. First Steps Towards an Event-Based Infrastructure for Smart Things. Ubiquitous Computing Workshop (PACT 2000), October 2000. [12] V. Jacobson. A New Way to look at Networking. Google Tech Talk, August 2006. [13] T. Koponen et al. A Data-Oriented (and Beyond) Network Architecture. In Proceedings of the 2007 Conference no Applications, Technologies, Architectures, and Protocols for Computer Communications, 2007. [14] T. Plagemann, V. Goebel, A. Mauthe, L. Mathy, T. Turletti, and G. Urvoy-Keller. From Content Distribution Networks to Content Networks – Issues and Challenges. Computer Communications Journal, 29(5):551–562, March 2006. [15] J. Giles. Internet Encyclopaedias go Head to Head. Nature, 438:900–901, 2005.



Software Architecture for Mobile Internet Service Platform
Web-based mobile robot platform for real-time exercises
Towards a service platform for mobile context-aware applications
Energy-Efficient Platform Designs for Real-World Wireless Sensing Applications
Towards Distributed Service Platform for Extending Enterprise Applications to Mobile Comput
A Service Platform for Internet-Telecom Services using SIPS
An Evolving Mobile E-Health Service Platform
SOA-based Integration of the Internet of Things in Enterprise Service
Security and Privacy in the Internet Era Bringing Real World to Classroom
A real-time computer vision platform for mobile robot applications
Case Study in Assessing Subjective QoS of a Mobile Multimedia Web Service in a Real Multi-a
An Evolving Mobile E-Health Service Platform