Angelo Corsaro, PhD – Chief Technology Officer, ADLINK
Gabriele Baldoni – Technologist, ADLINK
Early Internet of Things (IoT) applications adopted cloud-centric architectures where information collected from things is processed in a cloud infrastructure and decisions are pushed back from the cloud to things. While this architectural paradigm is suitable for a subset of Consumer IoT (CIoT), it quickly shows its limitation in the context of Industrial IoT (IIoT). More specifically, the following assumptions, at the foundation of cloud-centric architectures, are generally violated in IIoT applications:
Connectivity. Cloud-centric architectures assume that things are sufficiently often connected. While this is mostly true for IoT applications, it is far from being the common case in IIoT applications. As an example, autonomous agricultural vehicles, or robots in a smart farm are often deployed in locations with very poor connectivity.
Latency. Cloud-centric architectures assume applications can tolerate the latency associated with pushing data from things to the cloud, processing information on the cloud and eventually sending back some control information. This latency, is orders of magnitude higher of the reaction times required by several IIoT applications, such as, autonomous vehicles, smart factories and smart grids.
Throughput. Cloud-centric architectures assume that the throughput required to push data from things to the cloud may be massive when looking at the aggregate traffic, but it is generally composed by limited individual flows. In IIoT the situation is quite different as, often, there are data flows with high individual throughput and in several applications the aggregate volume is incredibly high. This makes it unfeasible or not very effective to stream this massive volumes of data to a data center.
Cost of Connectivity. CIoT commonly assume that the cost of connectivity is negligible. This stems from the fact that consumer pays for connectivity – either via their mobile data plan or their home internet connection. In IIoT the situation is completely different, it is the owner of the system that pays for connectivity. This cost is non-negligible in applications with large number of data streams, such as Smart Grids as well as for applications deployed in remote areas such as oil exploitation rigs that can only rely on expensive satellite communication where 1MByte of data can cost as much as $8!
Security. Cloud-centric architectures operate under the assumption that end-users are comfortable in giving away their data. While this may be true for CIoT applications, the situation is completely different in IIoT. Information is a strategic asset which the vast majority of companies, operating in an industrial, do not want to leave their premises.
Fog computing has emerged as an architectural approach to deal with the limitations exposed by cloud-centric architectures in the context of IIoT applications.
Early fog computing initiatives and demonstrations , focused on enabling cloud-less architectures by leveraging edge infrastructure. The main aim was exploiting relatively capable edge infrastructure to bring computing closer to were data was being produced and control needed to be actuated. Things were left out of the picture. In other terms we had moved from a cloud-centric to an edge-centric architecture but not necessarily toward an end-to-end solution. As a result, mist-computing ,  was evoked as the virtualisation of computing, storage and communication resources on things. Thus, for some time, instead of unifying the perspective, the community was segregating around the three tiers found in a typical CIoT and IIoT, in other terms the cloud, the edge and the things. The segregation was eventually resolved by the OpenFog Consortium Architecture Working Group in the vision paper , where the authors of this paper, along with the key companies driving fog computing agreed on the definition reported below. Fog computing. A system-level architecture that distributes computing, storage, control and networking functions closer to the users along a cloud-to-thing continuum  In other terms, fog computing aims at addressing the technological fragmentation existing today across the cloud the edge and things, by providing a unifying abstraction. This should sound completely logic as in general CIoT and IIoT applications will span across the three tiers and it would be ideal to have a unified set of abstractions to manage them.
How is Fog Different
A typical fog deployment is quite the opposite of a data center environment, as (1) nodes may not be extremely resourceful, (2) the system is highly asymmetric w.r.t. computational and connectivity capabilities, and (3) nodes have a high degree of churn which beside from failures and intermittent connectivity stems from mobility – consider for instance autonomous driving vehicles in a factory.
Additinoally key characteristic of fog environments is the need for supporting low reaction times, as well as the provisioning, in addition to VM and containers, of different kinds of deployable units, such as unikernels, microservices, and binary executables, some of which may need to run on real-time operating systems and leverage real-time networks such as Time Sensitive Networking (TSN) (IEEE 802.1) . Cloud-derived Infrastructure operates at a time scale that is incompatible with these kinds of systems. As an example the instantiation of a unikernel is dominated by OpenStack as opposed to the unikernel boot time. This clearly offsets the advantage of using unikernels for fast boot.
Introducing Eclipse fog∅5
As described above, the fog infrastructures proposed thus far in literature , , , , while being a step in the right direction, fall short in addressing the end-to-end unification of resources. The main limitations of previous solutions can be summarized as (1) deployment limited mostly to VM and/or Containers, (2) lack of support for real-time, (3) lack of mudularity with consequences on extensibility, and (4) limited security. Eclipse fog∅5 defines a set of abstractions to unify the compute, storage and communication fabric end-to-end and thus allow applications to be managed, monitored and orchestrated across the cloud to thing continuum. Before delving into the architectural details of fog∅5 let’s see the key abstraction that it uses in order to provision, manage and orchestrate, generic applications end-to-end.
The abstraction used by fog∅5 to provision, manage and orchestrate applications, or network functions, is the entity. An fog∅5 entity is either an atomic entity, such as a Virtual Machine, a container, a Unikernel, a binary executable, or a Directed Acyclic Graph (DAG) of entities (see Figure 1). Where the set of atomic entities supported by fog∅5 can be extended through plugins. Entities and Atomic Entities have a FSM that defines the legal state transitions. These FSM have been defined to be sufficiently generic to encompass essentially any kind of atomic entity ranging from a VM to a binary executable. Entities may define a deployment affinity w.r.t. to each other as well as with respect to compute, storage, I/O and accelerators such GPUs and FPGAs. The state machines that define the legal state transitions for entities and atomic entities are reported below.
The high-level architecture of fog∅5 is depicted in the diagram below. From this diagram, it can be seen how fog∅5 has been designed to run as either a process on a traditional OS or as a trusted service inside a Trusted Execution Environment (TEE). Where the TEE could be running on an hypervisor – as depicted in Figure 3 – or on bare metal, ideally on a trust zone such as those supported by ARM processors.
This deployment ensure that all communications with the external network are done through a run-time that has a very small surface of attack. Internal communication is performed through a VPN. This means that only the software running on the TEE can communicate with the external network, which, as such mediates communication between the internal and external network. This architecture, while not compulsory – as mentioned above – was motivated by the fact that regular operating systems, such as Linux and Windows, have an extremely large surface of attack, which makes them extremely hard to secure. Finally, the fog∅5 instance running on the TEE has only the plugins for managing the network and the hypervisor, along with a trusted implementation of the distributed store.
The core component of fog∅5 is the agent. The agent represents a manageable resource on the system, which from now on we’ll call a fog∅5 node. Each agent has two stores, one representing the actual state of the node and the other representing the desired state for the node. The actual state of a node can only be written by the agent running on that node, while the desired state can be written by anybody to cause state transitions, such as provisioning a VM, or a binary executable, and so on. Agents dynamically discover each other by leveraging the dynamic discovery provided by the distributed store, which in turn leverage the dynamic discovery provided by the data sharing layer which may be implemented via Data Distribution Service (DDS)  or zenoh . The set of functionalities supported by an agent is controlled by plug-ins. Different class of plug-ins exist for entities, network, OS, and so on. The agent ”simply” orchestrates the plug-ins state transitions.
## fog∅5 Plugins As mentioned above, fog∅5 leverages plugins to manage just about anything. If we take as an example the plug-in for atomic entities, these are responsible for implementing the state transition depicted in Figure 2 for a specific kind of deployment, unit, such as a Robotic Operating System (ROS) nodelet, a unikernel, a binary executable, a container or a VM. As another example, operating system plugin, provides an implementation of the primitives required by fog∅5 to operate on the given OS.
Eclipse fog∅5 and Fog Computing Requirements
Now that we have provided an overview of fog∅5, it is time to review how it addresses the key fog-computing requirements introduced earlier in this post.
Dynamic Discovery. fog∅5 dynamic discovery is provided by the distributed store, which in in turns, leverages the dynamic discovery provided by DDS or by zenoh. This dynamic discovery is implemented by decentralized algorithms that do not introduce dependencies on servers and facilitate zero-conf deployments.
HW and I/O Discovery. fog∅5 provides information on any kind of I/O and hardware accelerators available on the node. This information is managed by plugins and can be extended to provide access to standard API for dealing with embedded hardware and I/O such as the SGET eAPI.
Decentralized Architecture. fog∅5 has a decentralized architecture as a consequence of the decen-tralized key/value store used at its foundation.
Low Latency. As the distributed key/value store is implemented using either DDS or zenoh, the latencyfor distributing data on a LAN, is around 50 micro-seconds when using DDS and around 20 microseconds when using zenoh.
Low Resource Utilization. The fog∅5 agent takes around 3MB of memory on a 64-bit machine running Linux. This is for a fully functional agent with no additional external dependency. Notice also that, as the architecture is decentralized, there is nothing else beside the fog∅5 agent that needs to be deployed.
Generalized Constraint-based Deployment. fog∅5 uses a plug-in for allocation. The default, plugin takes into account computational resources, networking, I/O accelerators as well as affinities with respect to other entities.
Device Assignments. fog∅5 can assign any kind of device or I/O, such as a GPIO to an atomic entity.
Support for Constrained Devices. The term constrained devices may mean different things to different people. It should be clear from the number provided above that something like a Raspberry Pi is not a constrained target for us. We are able to target ARM-M and even less capable hardware, such as micro-controllers.
End-to-end Unification of Computing, Networking and Storage. As fog∅5 can run efficiently on high-end hardware as well as constrained things, it can manage the compute, storage and networking fabric end-to-end. Additionally, fog∅5 provides with the ability to configure end-to-end networks that span across cloud, edge and things.
Large Scale Deployment. Thanks to the dynamic discovery, large scale deployments are is simple and effective. The scalability of these deployments stems for the use of DDS and its dedicated infrastructure for large scale-deployments.
Extensibility. fog∅5 has a plugin architecture that allows for any aspect to be extended and customized.
Support for Real-Time OS. At the present time, fog∅5 can provision real-time OS and we are actively working on supporting TSN provisioning and management.
This post has provided an overview on the evolution of fog computing and summarized the key requirements that fog computing platforms need to address. These requirements highlight how, the main stream approach of adapting cloud-derived technologies fall short in addressing the most important characteristics of fog computing environments. The paper has thus introduced fog∅5 a novel fog computing infrastructure designed bottom up to address the key requirements of fog computing.
fog∅5 features a decentralized and plug-in-based architecture that allow for zero-conf deployment, low-latency, and extensibility. Our future work will focus on extending the applicability of fog∅5 to micro-controllers and providing consolidation and placement algorithms that perform dynamic resource optimization.
Finally, fog∅5 is an Eclipse IoT project and available at http://fog05.io
-  The data distribution service, 2017.
-  zenoh: The zero network overhead protocol, 2017.
-  Flavio Bonomi, Rodolfo Milito, Jiang Zhu, and Sateesh Addepalli. Fog computing and its role in the internet of things. In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing, MCC ’12, pages 13–16, New York, NY, USA, 2012. ACM.
-  D. Bruneo, S. Distefano, F. Longo, G. Merlino, A. Puliafito, V. D’Amico, M. Sapienza, and G. Torrisi. Stack4things as a fog computing platform for smart city applications. In 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pages 848–853, April 2016.
-  Angelo Corsaro. Cloudy, foggy and misty internet of things. In Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering, ICPE ’16, pages 261–261, New York, NY, USA, 2016. ACM.
-  S.K.Datta,C.Bonnet,andJ.Haerri. Fog computing architecture to enable consumer centric internet of things services. In 2015 International Symposium on Consumer Electronics (ISCE), pages 1–2, June 2015.
-  OpenFog Consortium Architecture Working Group. Openfog architecture overview. Technical report, Feb 2016.
-  Adrien Lebre, Jonathan Pastor, Anthony Simonet, and Frederic Desprez. Revising OpenStack to Operate Fog/Edge Computing infrastructures. In IEEE International Conference on Cloud Engineering, Vancouver, France, April 2017.
-  et al. M. Yannuzzi. A new era for cities with fog computing. IEEE Internet Computing, 21(2):54–67, Mar.-Apr. 2017.
-  J. S. Preden, K. Tammema ̈e, A. Jantsch, M. Leier, A. Riid, and E. Calis. The benefits of self-awareness and attention in fog and mist computing. Computer, 48(7):37–45, July 2015.
-  Seifeddine Nsaibiund Ludwig Leurs. Timesensitive Networking. atp edition, 58(10):40–47, 2016.
-  Shanhe Yi, Cheng Li, and Qun Li. A survey of fog computing: Concepts, applications and issues. In Proceedings of the 2015 Workshop on Mobile Big Data, Mobidata ’15, pages 37–42, New York, NY, USA, 2015. ACM.