An Energy-Efficient Spectrum-Aware Reinforcement Learning-Based Clustering Algorithm for Cognitive Radio Sensor Networks

Abstract

It is well-known that clustering partitions network into logical groups of nodes in order to achieve energy efficiency and to enhance dynamic channel access in cognitive radio through cooperative sensing. While the topic of energy efficiency has been well investigated in conventional wireless sensor networks, the latter has not been extensively explored. In this paper, we propose a reinforcement learning-based spectrum-aware clustering algorithm that allows a member node to learn the energy and cooperative sensing costs for neighboring clusters to achieve an optimal solution. Each member node selects an optimal cluster that satisfies pairwise constraints, minimizes network energy consumption and enhances channel sensing performance through an exploration technique. We first model the network energy consumption and then determine the optimal number of clusters for the network. The problem of selecting an optimal cluster is formulated as a Markov Decision Process (MDP) in the algorithm and the obtained simulation results show convergence, learning and adaptability of the algorithm to dynamic environment towards achieving an optimal solution. Performance comparisons of our algorithm with the Groupwise Spectrum Aware (GWSA)-based algorithm in terms of Sum of Square Error (SSE), complexity, network energy consumption and probability of detection indicate improved performance from the proposed approach. The results further reveal that an energy savings of 9% and a significant Primary User (PU) detection improvement can be achieved with the proposed approach.

1. Introduction

Technological advances in microelectronics have led to the widespread applications of wireless sensor networks (WSNs) in a variety of application areas. In general, wireless sensor nodes and many other wireless devices based on Wi-Fi, Zigbee and Bluetooth standards operate in unlicensed spectrum bands such as the Industrial Scientific and Medical (ISM) band which lack tight regulations. This leads to severe congestion in the useable unlicensed spectrum bands and causes harmful interference between the various wireless devices. On the other hand, licensed spectrum bands which are assigned to licensed users known as Primary Users (PUs) tend to become underutilized due to their fixed spectrum band allocation, as reported in the Federal Communications Commission (FCC) report [1]. This necessitates the need for a paradigm shift from the conventional inefficient spectrum allocation policy to a dynamic and more flexible spectrum access management.

Cognitive Radio (CR) is a new paradigm that has the potential to efficiently utilize the unused licensed spectrum bands, also known as spectrum holes, by dynamically allocating the spectrum holes to unlicensed users referred to as Secondary Users (SUs) without any harmful interference with PUs’ transmissions. Therefore, the main motivation for CR is dynamic access to temporal and spatial spectrum holes [2]. Spectrum sensing is the main fundamental function of CR for spectrum band exploration to identify spectrum holes and to protect PUs from harmful interference. Two performance metrics namely, probability of detection P_d and probability of false alarm P_f are used to measure the reliability of spectrum sensing techniques for discovering the availability or otherwise of spectrum holes. However, wireless propagation impairments such as multi-path fading, receiver uncertainty, shadowing and interference in wireless channels degrade the performance of PU detection techniques [3].

Cooperative spectrum sensing is a promising approach to overcome these problems [4]. The approach involves coordinating multiple CRs to share their local sensing results and make a collective decision about spectrum hole availability. It also improves the probability of PU detection through exploration of multi-users’ sensing diversity. A Fusion Centre (FC) performs decision fusion on the sensing results obtained and makes a global decision on the spectrum holes’ status. Although cooperative spectrum sensing yields better sensing performance, it also increases communications overhead, incurs in high energy consumption as well as extra sensing and reporting delays, particularly in large-scale networks such as CR-WSN. These problems can be minimized by logical grouping of multiple SUs to form a cluster.

CR-WSN is a network of dispersed wireless sensor nodes embedded with cognitive radio capability which enable them to dynamically access unused licensed spectrum bands for data transmission while performing conventional wireless sensor nodes’ tasks [5]. CR-WSN offers several potential benefits to a wide range of applications domains and has been proposed as one of the most promising technologies to address spectrum access and utilization challenges in WSN [6,7]. For instance, when multiple conventional sensor nodes attempt to simultaneously transmit data through the overcrowded unlicensed spectrum bands, the transmitted packets may not get to the destination due to packet collisions. This not only leads to excessive network power consumption as a result of packet retransmissions, but also increases the probability of packet collisions which significantly affect the communication reliability of the network [8]. Although, cognitive radio sensor nodes can dynamically access multiple unused licensed channels for data transmission in order to mitigate this challenge, the additional task of opportunistic access to unused licensed channels through spectrum sensing incurs a significant energy cost. This means that CR-WSN inevitably consumes much more energy than conventional WSN due to the cognitive capability.

Generally, cognitive radio sensor nodes are characterized by limited energy, constrained storage and processing resources, which are inherited from conventional wireless sensor nodes. Therefore, the main challenges in CR-WSNs are energy efficient communications to extend the lifetime of the network and PU protection from unlawful interference.

Network clustering involves partitioning the network into logical groups of nodes that form clusters, each cluster comprises of a clusterhead (CH) while the none clusterhead nodes are referred to as Member Nodes (MNs). The CH may serve as a central point to all nodes in the cluster, and it performs various tasks such as data aggregation and spectrum sensing coordination. In addition, it also provides inter-cluster communications by communicating with neighboring CHs and a Base Station (BS). The MN detects events and communicates its data to the associated CH through intra-cluster communications through either single-hop or multi-hop routing.

The network clustering process generally involves three phases: initialization, setup and maintenance phases, which gives the main distinction among the various clustering algorithms. The initialization phase can either be centralized or distributed. The setup phase involves emergence of CHs based on either pre-defined metric functions or random selection, it also involves formation of MNs in the cluster where each MN joins its respective cluster either by default or based on some metric function. Finally, the maintenance phase deals with rotation of nodes’ roles and re-clustering of the network when a pre-defined condition is reached or at the beginning of every round.

Clustering of a network has several benefits and it has been widely explored in conventional wireless sensor networks; they are firstly to achieve network scalability [9], and at the same time prolong the lifetime of a network [8]. However, its application in CR-WSN to enhance PU protection has not been fully explored [10]. Existing clustering algorithms mainly focus on routing [11] and energy consumption issues in conventional WSNs [12], and only a few have attempted to address both energy consumption and spectrum sensing performance issues in CR-WSN.

Therefore, conventional clustering algorithms for WSNs or mobile ad hoc networks may not be suitable for CR-WSN due to the dynamic nature of the channels. This necessitates the need for a novel clustering algorithm that will address both energy issues and spectrum holes detection issues in CR-WSN. Network clustering to support many cognitive radio tasks such as dynamic channels access, cooperative sensing and routing has been extensively discussed in [13]. Cognitive radio-based network clustering requires additional conditions for grouping of nodes based on common vacant channels detected in a temporal and spatial neighborhood [14]. In such a network, sensor nodes within a cluster are require to have at least one common vacant channel between the transmitter and the receiver for communication. The dynamic nature of the environment which is influenced by the PU activities necessitates the need for spectrum aware clustering schemes.

Spectrum-aware clustering schemes in cognitive radio networks has received considerable attention in recent times. Network scalability and heterogeneity challenges have been well investigated and addressed such as in [15]. The scheme is based on distributed coordination approach where SUs construct groups in accordance with common vacant channels locally detected. A spectrum-aware routing solution for cognitive radio described in [16] selects routes that offer the highest spectrum availability and computes its long-term routing metrics to balance between short-term route performance and long term route satiability. The scheme mainly addresses channelization and dynamic variation issues in cognitive radio routing protocol to effectively utilize unused licensed channels. The spectrum aware clustering scheme in [14] is mainly driven by an event which requires a temporal cluster. The scheme uses nodes’ local position in respect to the event and sinks to select eligible nodes for clustering. It then elects a clusterhead among the eligible nodes based on channel availability, node degree and distance to sinks in the neighborhood. The authors in [17] proposed a centralized Groupwise Spectrum-Aware (GWSA) clustering algorithm; it first creates a proximity matrix for all nodes, determines from the matrix the global minimum distance between pairs and then merges the nearest clusters that satisfy the Groupwise constraints in each iteration until an optimal number of clusters that minimizes network-wide energy consumption is achieved.

However, this approach suffers from network instability because PU arrival may cause re-clustering of the whole network and also its high computational complexity which increases proportionately along with the size of the proximity matrix limits its practical implementation in a large-scale network. The algorithm described in [18] minimizes the network instability problem such that re-clustering involves only nodes that detect the PU arrival while network topology of other nodes remains intact. Furthermore, it determines local minimum distance between neighboring cluster pair and merges multiple nearest cluster pair at a single time to increase the convergence rate. Although the algorithm achieves relatively much less computation complexity, its performance is relatively inferior to the GWSA. Unlike [17,18] our proposed algorithm not only tends to minimize network energy consumption, but also improves spectrum hole detection, by way of exploring multi-user sensing diversity through cooperative spectrum sensing which is essential for dynamic spectrum access in cognitive radio.

The reinforcement learning (RL) technique has been applied to many existing works related to cognitive radio, but they were mainly applied to spectrum sensing [2], cooperative sensing [3,19], spectrum sharing [20], channel sensing [21,22] and dynamic channel access [23,24,25]. For example, the authors in [3,19] applied RL to enhance cooperative gain and mitigate cooperative overhead in cognitive radio. The approach eliminates correlated and unreliable cooperative neighboring SUs from cooperation and determines optimal set of cooperative SUs that minimize spectrum sensing delays and control channel traffic. In a bid to improve channel sensing performance, the authors in [26] used channel achievable and channel availability as the basis for determining optimal channel sensing order by applying a low complexity RL algorithm. The approach in [2] achieved energy efficient spectrum sensing by exploration of sensing assignment and exploitation of high throughput frequency bands. Even though the existing RL approaches in the literature made valuable contributions to CR-WSN, none of them considered RL for network clustering

In this paper, we propose an Energy Efficient Spectrum Aware clustering algorithm based on Reinforcement Learning (EESA-RLC) to enhance spectrum hole detection and minimize network energy consumption in CR-WSNs. Reinforcement learning is a machine learning technique that allows an agent to interact with its operating environment and learn an optimal policy that maximizes cumulative rewards [27]. The operating environment can be formulated as a Markov Decision Process (MDP) comprising actions, state of the system, transition rewards, transition probability, performance metric and policy [28]. The agent, which in this case is the SU, detects vacant licensed channels through channel sensing, imposes pairwise constraints to select a clusterhead among the neighboring clusterheads, cooperates with other member nodes in the cluster to determine channel availability, and then chooses an optimal policy that enhances spectrum hole detection and minimizes network energy consumption. The agent employs a Temporal Differences (TD) learning technique [29] to learn from the neighboring clusterheads, evaluates its local decision accuracy, distance to the clusterheads which translates into energy consumption and selects an optimal action policy that optimizes its performance in determining an optimal clusterhead. Although RL is a well-known machine learning technique and has been extensively applied to many fields such as cooperative sensing [3,30], channel sensing sequence [26,22], energy efficient communication [2,31], its application in clustering algorithm is still at the infancy stage. Therefore, our contributions in this paper can be summarized as follows:

• We propose a novel energy-efficient clustering algorithm that is aware of the dynamic radio environment and allows member nodes to learn an optimal policy for choosing optimal clusters based on local decision accuracy and energy consumption for cooperative sensing and data communication.
• The proposed algorithm implements pairwise constraints in spectrum-aware clustering such that only SUs with at least one common vacant channel with a clusterhead and within the clusterhead’s one hop radio range can form a cluster.
• We model network energy consumption, cooperative channel sensing, inter-cluster and intra-cluster data communication energy consumptions and determine an optimal number of network clusters that minimizes network energy consumption.
• We show the performance improvements of the proposed clustering algorithm over Groupwise constraint-based algorithms [17,18] in terms of energy efficiency, channel sensing performance and computational complexity, which make it more attractive for resource constrained devices such as CR-WSNs. In addition, the algorithm eliminates network instability due to re-clustering when the SUs detect PUs’ arrival.

2. System Model

This section describes the system model and parameters adopted for this research. The network is assumed to be static, consisting of $N$ non-mobile homogenous fully functional cognitive radio sensor nodes capable of performing complex tasks. The number of SUs in the network exert a significant impact on energy consumption and on sensing performance. For instance, in a fixed size cluster, the cooperative probability of detection increases along with the increase in the number of cooperative SUs. The network area is partitioned into $q$ clusters, each cluster can be seen as a small cell network comprising a clusterhead and a few member nodes, as shown in Figure 1. Partitioning the network area into clusters has a significant effect on the network energy consumption. If the number of clusters in the network is too small, spectrum sensing, reporting and data communication consume a large amount of energy owing to large number of member nodes per cluster. On the other hand, if the number of clusters is too many, the number of member nodes would be too small and hence the energy consumption per cluster would be low but a large amount of energy would be consumed for inter-cluster communication. Therefore, the optimal number of clusters is extremely important.

Figure 1. Clustered cooperative channel sensing.

Figure 1. Clustered cooperative channel sensing.

The nodes are uniformly distributed in a two-dimensional square area ${\text{N}}_{\text{A}}$ of $\text{L}$ × $\text{L}$ square meters and each node is battery powered. This means sensor nodes’ energy cannot be recharged, therefore, nodes’ energy consumption need to be minimized to extend the lifetime of the network. Each node can operate either as a clusterhead or member node. The member nodes MNs sense a set of licensed channels to detect vacant channels, report local sensing decisions to clusterheads for cooperative decision-making and also sense the environment to detect events. The clusterheads perform additional tasks which include decision fusion on the sensing results, controlling access to free channels for data communication and coordinating channel sensing. These additional tasks drain more energy from the battery of the clusterhead, therefore the role of clusterhead will be reassigned to other member nodes within a cluster when the energy depletes below a threshold. It is further assumed that all member nodes lie within the radio range of their respective clusterheads $(d<R_{max})$ and communicate directly with the clusterheads in a single-hop manner. This means that member nodes require only low transmissions power and at least one common vacant channel to communicate their data to clusterheads since the distance between them is short.

The operation of the SUs are divided into time slots $\tau$ of durations, ${\tau }^{cs}$ is the time allocated for sensing channels, ${\tau }^{rp}$ is the time duration for reporting the results, and ${\tau }^{dt}$ is duration over which the SU can access the free channel for data transmission. There are $n_z$ heterogeneous licensed channels in which each channel may exhibit different bandwidth and channel conditions. Larger bandwidth requires longer sensing time which translates into higher energy consumption and poor channel conditions resulting in inaccurate local decisions. An energy detection technique is employed to detect the presence of PUs on the licensed channels, since the PUs’ signals are assumed to be unknown, coupled with the unique resource constraint features of the CR-WSN which requires less complex spectrum sensing techniques.

3. The Proposed Energy Efficient Spectrum Aware Reinforcement Learning Based Clustering (EESA-RLC) Algorithm

This section presents modelling and algorithms for our energy-efficient reinforcement learning-based clustering scheme. The algorithms basically comprise of initialization, set-up and coordination phases.

The initialization phase precedes the reinforcement learning process; it involves election of clusterheads. Each SU senses a predefined set of channels to detect the presence or absence of PUs in the channels, computes its clusterhead probability $p_n^{ch}$ based on the number of vacant channels detected ${\phi }_n$ and residual energy ${\text{E}}_n^{\text{Re}}$, as well as the required percentage of clusterheads $\psi$ (e.g., 5%) for the network. The clusterhead probability is the probability of a sensor node to become a clusterhead. The main goal is to ensure that all SUs in the network are covered by a set of clusterheads at the initial stage, so that member nodes can directly communicate with clusterheads within their radio range via a single-hop while the clusterheads communicate with the BS through a single-hop or multi-hop fashion. The clusterhead probability $P_{ch}$ for secondary user $S{U}_n$ can be expressed as:

$$p_n^{ch}=\frac{{\text{E}}_n^{\text{Re}} {\phi }_n \psi }{{\text{E}}_n^{\text{mx}} n_z}$$

(1)

where ${\text{E}}_n^{\text{mx}}$ denotes the reference maximum energy of the SU when fully charged. This clusterhead probability is similar to the HEED protocol described in [32].

The clustering initialization process begins at time ${\tau }_{ini}$. Each SU determines its clusterhead probability and compares it with a given threshold $\psi <1$. If its clusterhead probability is greater or equal to the given threshold $p_n^{ch}\ge \psi$, then the SU emerges as a tentative clusterhead and then broadcasts an advertisement packet $A_{pkt}$ comprising its ID and clusterhead probability. This means that SUs with the highest probability are more likely to emerge as the tentative clusterheads. SUs with clusterhead probability less than the threshold $(p_n^{ch}<\psi )$ hearing the clusterheads announcement withdraw from competing and wait for the final clusterheads announcement by the BS. These SUs are more likely to remain as member nodes while the other in the set perform $N_{ir}=1/\xi$ maximum number of iterations and compete for the role of clusterhead after expiration of announcement waiting period ${\sigma }_s$. Where ξ denotes percentage of the threshold which can be set to a value less than one $\text{ξ}<1$.

In each iteration $i_{tr}$, $i_{tr}\le N_{ir}$, each of the SUs increases its clusterhead probability $P_{ch}$ by ξ (e.g., ξ = 10% of $\psi$ and compares the updated clusterhead probability with the given threshold. If the clusterhead probability is greater than or equal to the threshold, then it terminates the iteration and broadcasts an advertisement packet. Otherwise, it proceeds to the next iteration. The set of SUs with least number of iterations would emerge as the tentative clusterheads, then after expiration of announcement waiting period ${\sigma }_s$ the other sets that follow them would be elected. This process continues until all nodes are covered by the clusterheads. This means that after the emergence of the first set of clusterheads, each subsequent emergence of sets of clusterheads would be delayed by some duration depending on the number of iterations. The BS selects an optimal number of clusterheads among the tentative clusterheads and broadcasts the list.

The set-up phase mainly deals with cluster formation, based on the advertisement packet $A_{pkt}$ received from multiple neighboring clusterheads $C{H}_j|j=1, 2, 3,\dots ,n_{hg}$ which also denotes clusters $c{l}_j|j=1, 2, 3,\dots ,n_{hg}$, hence clusterhead and cluster will be used interchangeably in this paper. The SU learns the energy consumed and local decision accuracy for each of the clusters by executing model-free reinforcement learning and then selects an optimal cluster $c{l}_j^{*}$ that minimizes energy consumption and enhances spectrum holes detection. During the learning process, the SU senses set of licensed channels $S_{ch}^t=\left\{C{h}_z|z=1, 2, 3,\dots ,n_z\right\}$ at every episode $t$, sends its local decision $L{D}_i^t= \left\{D_z|z=1, 2, 3,\dots ,n_z\right\}$ to the clusterhead $C{H}_j$ for the final cooperative decision $C{D}_j^t= \left\{D_z|z=1, 2, 3,\dots ,n_z\right\}$ and then compares its local decision $L{D}_i^t$ with the cooperative decision $C{D}_j^t$ to determine the local decision accuracy $LD{A}_j^t$ in respect to cluster $c{l}_j$. It also determines the energy consumption for communicating data $E_{dcost,j}^t$ and cooperative sensing $E_{pcost,j}^t$. In addition, during the process, favourable clusters which offer minimum energy consumption and better spectrum hole detection would be selected, while excluding the less favorable clusters.

The maintenance phase involves coordination of cluster members. In this phase, the clusterheads specify the set of channels $S_{ch}$ to be sensed based on their availabilities and control access to the free licensed channels for data communication. Upon energy depletion of any clusterhead, the clusterhead initiates re-clustering process and a new clusterhead would emerge among the member nodes.

The main objective of the algorithm is to achieve an optimal policy for selecting optimal cluster or clusterhead that satisfies the pairwise constraint conditions, minimizes cooperative channel sensing energy consumption and data communication energy consumption while enhancing spectrum hole detection.

Let $EH=\left\{C{H}_j|j=1, 2, 3,\dots ,hn\right\}$ denote a set of clusterheads and $H_n=\left\{C{H}_j|j=1, 2, 3,\dots ,n_{hg}\right\}$ denote a set of neighbouring clusterheads such that $H_n\subset EH$. And let $C{L}_n=\left\{c{l}_j|j=1, 2, 3,\dots ,n_{hg}\right\}$ and $E_{TC}^{set}\left(B_{dt},d\right)=\left\{E_{TC,j}\text{|}j=1, 2, 3,\dots ,n_{hg}\right\}$ denote the corresponding set of neighbouring clusters and set of energy consumed for transmitting $B_{dt}$-bits data packet to the respective clusterheads at distance $d$ and for cooperative sensing of set of channels $S_{ch}=\left\{C{h}_z|z=1, 2, 3,\dots ,n_z\right\}$. Each cluster $c{l}_j$ consists of a clusterhead $C{H}_j$ and member nodes $M{N}_i\text{|}i=1, 2, 3,\dots ,m_n\text{]}$ such that $c{l}_j=\left\{C{H}_j,M{N}_i|i=1, 2, 3,\dots ,m_n\right\}$. If $M_{V,i}=\left\{C{h}_z|z=1, 2, 3,\dots ,n_{mv}\right\}$ and $H_{V,j}=\left\{C{h}_z|z=1, 2, 3,\dots ,n_{hv}\right\}$ denote sets of vacant channels detected by the member node $M{N}_i$ and the selected clusterhead $C{H}_j$, respectively, then the problem of finding optimal clusterhead $C{H}_j^{*}$ can be formulated as a Markov Decision Process while the energy minimization problem can be formulated as pairwise constraint Sum of Square Error (SSE) minimization problem subject to one hop transmission constraint. This is given as:

$$\arg min{\displaystyle \sum }_{j=1}^q{\displaystyle \sum }_{i=1}^{m_n}{d}^{\beta }\left(i,j\right)$$

(2a)

Subject to:

$$\begin{array}{c}\text{C1:}{M}_{V,i}\cap H_{V,j}\ne \varnothing \\ \text{C2:}\begin{array}{c}\max d\left(i,j\right)<R_{max}\\ M{N}_i,C{H}_jϵc{l}_j\end{array}\end{array}$$

(2b)

To achieve this, Q-learning would be adopted due to its model-free capability and the pairwise constraint would be applied on the member nodes and the clusterheads during the clustering.

3.1. Pairwise Constraint Clustering

The concept of pairwise constraints has been widely implemented in many clustering algorithms such as k-means [33] and complete link [34] clustering to impose must-link and cannot-link constraints on pairs of nodes during the clustering as illustrated in Figure 2. The must-link constraint forces pair nodes $n_a$ and $n_b$ to be placed in the same cluster, while the cannot-link constraint disallows pair nodes $n_a$ and $n_b$ to be placed in the same cluster [17]. This significantly influences the outcome of the clustering, since pair nodes with common links usually belong to the same cluster, while those without common links belong to different clusters.

Figure 2. Illustration of must-link and cannot-link constraints.

Figure 2. Illustration of must-link and cannot-link constraints.

Therefore, the pairwise constraint concept can also be implemented in a spectrum-aware clustering algorithm, which can be explained by comparing it with the groupwise constraint method employed in [17,18] for spectrum-aware clustering. Unlike conventional WSN clustering schemes, spectrum-aware clustering schemes require each node to sense the spectrum band and detect spectrum holes that can be used for data communication. In addition, each clusterhead must have at least one common available channel with its member nodes since each member node transmits its data directly to the clusterhead without intermediary nodes. To highlight the difference between pairwise and groupwise constraints, we consider a clusterhead $C{H}_j$ and three member nodes $M{N}_1$, $M{N}_2$, $M{N}_3$ that operate on a set of licensed channels $S_{ch}=\left\{C{h}_z|z=1, 2, 3, \dots , n_z\right\}$ as shown in Figure 3. The numbers beside them represent the vacant licensed channels detected by the respective member node; a dotted line between them indicates a cannot-link constraint, while a solid line indicates a must link constraint as illustrated in Figure 2. The figure shows that member node $M{N}_1$ shares channel $C{h}_2$ with $M{N}_2$ and $C{H}_j$, member node $M{N}_2$ shares channel $C{h}_2$ and channel $C{h}_3$ with $C{H}_j$ in addition to member node $M{N}_1$, while member node $M{N}_3$ shares only channel $C{h}_1$ with $M{N}_1$. Based on this scenario, the pairwise constraint imposes a must-link constraint on member nodes $M{N}_1$ and $M{N}_2$ to form a cluster with clusterhead $C{H}_j$ because they all share a vacant channel ($C{h}_2$). It also imposes a cannot-link constraint on $C{H}_j$ and $M{N}_3$ so that the member nodes cannot form a cluster since member node $M{N}_3$ has no common vacant channel with $C{H}_j$ even though it shares a vacant channel with member node $M{N}_1$. On the other hand, the groupwise constraint imposes cannot-links on all the four nodes, including $C{H}_j$, disallowing them from forming a cluster, because the nodes share no common vacant channel. While a pairwise constraint requires at least one common vacant channel for pair nodes of clusterhead and member node to form a cluster, a groupwise constraint requires at least one common vacant channel for all nodes in a group to a form cluster.

Figure 3. Illusration of pairwise and groupwise constraints.

Figure 3. Illusration of pairwise and groupwise constraints.

3.2. Cooperative Channels Sensing

Spectrum sensing is a key function of cognitive radio for determining licensed channel occupancy; this is done by detecting the existence of PUs in the channels. Energy detection technique has been commonly employed to detect the existence of PUs’ signals in the spectrum bands by measuring energy of the received signal waveform over a specified observation time. The received signal is first filtered by a Band Pass Filter (BPF) to limit the noise bandwidth. The filtered output signal of bandwidth $B_w$ is converted to discrete samples $N_o$ by an Analogue-to-Digital Converter (ADC) and then passed through an integrator for an observation interval $\tau$. The final average energy of the observed samples $N_o$ from the output of the integrator $Y={\displaystyle \sum }_{\tau =1}^{N_o}{\left|y\left(\tau \right)\right|}^2$ is compared with a threshold λ to determine the existence or otherwise of a PU signal [4]. If $Y<\lambda$, then a PU’s signal is absent and the channel is considered available, otherwise, a PU’s signal is considered to be present and the channel is being occupied. Thus, the received signal at the SU $Y$ can be expressed as [4]:

$$Y= \{\begin{array}{c}x\left(\tau \right), H_o\\ z\left(\tau \right)+x\left(\tau \right), H_1\end{array}$$

(3)

where $x\left(\tau \right)$ denotes zero-mean Additive White Gaussian Noise (AWGN), $z\left(\tau \right)$ denotes the received signal waveform. $H_o$ denotes the null hypothesis which indicates the absence of a PU’s signal, while $H_1$ denotes a hypothesis which indicates the presence of a PU’s signal. Thus, the test statistics $Y$ from the output of the integrator follow a chi-square distribution and can be approximated to a Gaussian distribution using central limit theorem, which when the number of samples is large given as [29]:

$$Y\approx \{\begin{array}{c}\mathcal{N}\left(N_o{\sigma }_n^2, 2n{\sigma }_n^4\right), H_0 \\ \mathcal{N}\left(N_o\left({\sigma }_n^2+{\sigma }_z^2\right), 2N_o{({\sigma }_n^2+{\sigma }_z^2)}^2\right), H_1\end{array}$$

(4)

where ${\sigma }_n^2$ denotes received the noise signal’s variance and ${\sigma }_z^2$ denotes the received signal’s variance $z\left(\tau \right)$. Optimal PU detection can be achieved through a Maximum A Posteriori (MAP) scheme which models the PU behaviors into On and Off states using a two-state Markov chain that has been widely adopted [35,36,37]. Therefore, the probability of detection $P_d$ which suggests the presence of a PU in the considered channel can be expressed as:

$$\begin{array}{l}{P}_d(\mathrm{\lambda })=P_r\left[Y>\mathrm{\lambda }\backslash H_1\right]P_{on}\\ =\mathrm{\mathbb{Q}}\left(\frac{\mathrm{\lambda }-2B_{w}T\left({\sigma }_n^2+{\sigma }_z^2\right)}{\sqrt{4B_{w}T{({\sigma }_n^2+{\sigma }_z^2)}^2}}\right) . P_{on}\end{array}$$

(5)

Similarly, the probability of false alarm $P_f$ which falsely indicates the presence of a PU’s signal in the considered channel can be expressed as:

$$\begin{array}{l}{P}_f\left(\mathrm{\lambda }\right)=P_r\left[Y>\mathrm{\lambda }\backslash H_0\right]P_{off}\\ =\mathrm{\mathbb{Q}}\left(\frac{\mathrm{\lambda }-2B_{w}T{\sigma }_n^2}{\sqrt{4B_{w}T{\sigma }_n^4}}\right). P_{off}\end{array}$$

(6)

where $\mathrm{\mathbb{Q}}(.)$ is the generalized Marcum Q-function, $P_{on}$ and $P_{off}$ are probabilities that the channel is in busy or idle states respectively.

Cooperative spectrum sensing enhances the PU detection through exploitation of SUs’ observed signals spatial diversity. Each of the $M{N}_i$ senses set of channels $S_{ch}=\left\{C{h}_z|z=1, 2, 3,\dots ,n_z\right\}$, makes a local decision $L{D}_{i }= \left\{D_z|z=1, 2, 3,\dots ,n_z\right\}$ on the existence of PUs in the channels or otherwise and then reports its sensing result to the cluster head $C{H}_j$ for decision fusion and final cooperative decision $C{D}_{\text{j}} = \left\{D_z|z=1, 2, 3,\dots ,n_z\right\}$. Local decision $D_z=0$ indicates the presence of a PU’s signal in the observed channel $C{h}_z$, while $D_z=1$ denotes the absence of a PU’s signal in the considered channel $C{h}_z$. It is assumed that the channel between $M{N}_i$ and $C{H}_j$ is a perfect channel since the distance between them is short. The clusterhead $C{H}_j$ employs “M-out-of-N majority” decision counting rule fusion to determine the existence of PU in the channels $S_{ch}=\left\{C{h}_z|z=1, 2, 3,\dots ,n_z\right\}$ and then broadcasts the outcome. The final cooperative decisions $C{D}_j$ based on this rule indicates the presence of a PU’s signal in the channel when $\ell$out of $m$ sensing results indicate the presence of a PU’s signal in the channel [38]. This implies that when the number of cooperative member nodes $c{m}_n$ that report presence of PU’s signal in the considered channel is greater than or equal to half of the total number of cooperative member node ($c{m}_n\ge m_n/2)$, then the final cooperative decision indicates the presence of a PU in the observed channels [39]. Otherwise it indicates the absence of a PU and hence the availability of the observed channels. Thus, the cooperative probability of detection $Q_{d,j}$ is given as [30]:

$$Q_{d,j}={\displaystyle \sum }_{ \ell =n}^m\left(\begin{array}{c}m\\ \ell \end{array}\right){\text{P}}_d^{\ell }{\left(1-P_{\text{d}}\right)}^{m-\ell }$$

(7)

3.3. Cognitive Radio Wireless Sensor Network Energy Consumption Model

Network energy consumption for CR-WSNs mainly comprises of energy consumptions for vacant channels detection, event sensing, data processing, and communication. The energy consumption for detecting vacant channels is the energy consumed for cooperative channels sensing. The event sensing energy consumption is attributed to energy consumed for the sensing event while the data processing energy is attributed to energy consumed for data logging. The energy consumption for data transmission is attributed to energy consumed for intra-cluster and inter-cluster data communications.

Several attempts have been made to model wireless sensor network energy consumptions. The most widely adopted models [40,41,42] have combined the impact of the external radio environment and the sensor node’s communication hardware together. According to these models, energy consumption for transmitting a unit bit of data depends largely on the distance between the transmitting and the receiving nodes. However, the outcome of experimental measurements conducted in [43] is inconsistent with the widely adopted simplified models. In addition, it is revealed that separating energy consumption of each hardware component from the external radio environment may lead to a more realistic energy consumption model [44]. The authors in [45] proposed a comprehensive node power consumption model which considered other energy consumption sources that were ignored in the previous models such as sensor sensing and sensor logging. However, these approaches are specifically developed for conventional sensor nodes without due consideration of the cognitive radio aspect. Therefore, accurate estimation of CR-WSN life expectancy requires a realistic network energy consumption model that incorporates channel sensing energy consumption as well.

The main components of a cognitive radio sensor node are event sensing, data processing, communication and cognitive radio units as shown in Figure 4. The event sensing unit monitors the environment and generates signal traffic whenever an event is occurred. The processing unit processes the data while the communication unit transmits the data to desire sink over a free licensed channel and also receives data. The cognitive radio module detects set of unused licensed channels and then accesses the most suitable channel to communicate the data.

Figure 4. Block diagram of a cognitive radio sensor node.

Figure 4. Block diagram of a cognitive radio sensor node.

3.3.1. Event Sensing Unit Energy Consumption

The event sensing unit interacts with the physical environment to detect an event and then convert the physical signals to digital signals. Signal sampling, physical signal conversion to electrical signals and analogue signal to digital signal conversion are the main sources of energy consumption in the event sensing unit [45]. Let $T_{ss}$ denotes the event sensing duration, $P_{ss}$ denotes power required for the event sensing activity which includes event detection and signal conversion and $B_{ss}$ denotes the bits packet. The energy dissipation for event sensing activity for $B_{ss}$bits packet is given as:

$$E_{ss}\left(B_{ss}\right)=P_{ss} T_{ss} B_{ss}$$

(8)

3.3.2. Processing Unit Energy Consumption

The processing unit executes functions such as sensor data logging, data aggregation and processing. Energy consumption for sensor data logging is due to amount of energy consumed for reading and writing a packet of data into the memory. Let $P_{rd}$ denote the power consumption for reading a packet from the memory, $P_{wr}$ denotes power consumption for writing a packet into the memory then, the energy consumption for data logging can be expressed as [45]:

$$E_{log}\left(B_{lg}\right)=B_{lg}\left(P_{rd}+ P_{wr}\right)$$

(9)

Energy consumption for data aggregation and processing are mainly derived from microcontroller energy losses which occur as a result of switching and leakage currents. The total energy consumption for processing or aggregating $B_{ap}$ bits of data packet can be expressed as [46]:

$$E_{ap}\left(B_{ap},N_{cy}\right)=B_{ap}{N}_{cy}{C}_{av}{V}_s^2+B_{ap}{V}_s\left(I_o{e}^{\frac{V_s}{n_p{V}_t}}\right)\left(\frac{N_{cy}}{f}\right)$$

(10)

where $N_{cy}$ denotes the number of clock cycles per task, $I_o$ denotes the leakage current, $C_{av}$ denotes average number of capacitance switches per cycle, $f$ denotes the frequency of the sensor, $n_p$ denotes a constant parameter defined by the processor, $V_s$ and $V_t$ denote the source and terminal voltage, respectively [45].

3.3.3. Cognitive Radio Unit Energy Consumption

The CR unit senses the licensed channels and detects vacant channels that can be used for data communications. Energy consumption for vacant channels detection comprises energy consumption for sensing sets of channels and reporting local decisions as well as receiving final cooperative decisions [47]. Energy consumption for sensing sets of channels ${\text{E}}_{\text{cs}}$ comprises of energy consumed for listening over the channels and receiving $N_o$ observation samples, as well as energy required to process the signal samples (modulation, signal shaping etc.) and make local decisions. If $P_{ed}$ denotes the energy detector’s circuit power consumption and $E_{sp}$ denotes the energy consumption for processing the received $N_o$ signal samples, then energy dissipation for sensing $n_z$ sets of channels can be expressed as:

$${\text{E}}_{\text{cs}}\left(n_z,T_{cs}\right)={{\displaystyle \sum }}_{m=1}^{ n_z}\left(T_{cs} P_{ed}+{\text{E}}_{sp}\right)$$

(11)

This suggests that the energy consumption for channel sensing is a function of channel sensing duration $T_{cs}$ and it increases along with an increase in the number of channels $n_z$. Minimum energy consumption can be achieved with minimum channel sensing time but accurate results may not be obtained. The Nyquist sampling theorem suggests that the sample frequency $f_s$of the received signal samples ($N_o=2T_{cs}{B}_w)$ must be at least twice the bandwidth ($f_s\ge 2B_w$) [29]. Let $P_{dt}$ denote the target probability of detection, $P_{ft}$ denotes target probability of false alarms, and ${\delta }_s$ denotes the average of the PU’s SNR received on the channel. The channel sensing time $T_{cs}$ can be expressed as [37]:

$$T_{cs}=\frac{1}{{{\delta }_s}^2{f}_s}{\left(Q^{-1}\left(P_{ft}\right)-Q^{-1}\left(P_{dt}\right)\sqrt{2{\delta }_s+1}\right)}^2$$

(12)

In cooperative sensing, each member node $M{N}_i$ senses sets of channels to detect vacant channels, makes local decisions on the existence of PU and then reports its result to the FC which is the clusterhead $C{H}_j$ for the final cooperative decision.

Let $d_{i,j}$ denote the Euclidian distance between $M{N}_i$and $C{H}_j$, $E_{ec}$ denotes the energy consumption for running the radio electronics of $M{N}_i$ and $E_{am}$ denotes the energy consumption for amplifying the signal to be transmitted to $C{H}_j$so as to maintain an acceptable SNR level. Then the energy cost $E_{rp}$ for reporting $B_{ld}$-bits packet of local decisions $\text{to the }C{H}_j$ can be expressed as:

$${\text{E}}_{rp}\left(B_{ld},d_{i,j}\right)= B_{ld}\left(E_{ec}+E_{am} d_{i,j}\right)$$

(13)

The energy cost for receiving the $B_{ld}$ bits packet of final cooperative decision broadcast by the $C{H}_j$ after performing a decision is mainly determined by the number of bits in the packet and energy consumed for running the radio electronics circuitry. Therefore, the energy consumption for receiving $B_{ld}$-bits of broadcasted packet can be expressed as:

$${\text{E}}_{rx}\left(B_{ld}\right)= B_{ld}{E}_{ec}$$

(14)

Therefore, the energy consumed by member node $M{N}_i$ is the energy consumption for cooperative channel sensing which comprises the energy consumption for sensing the set of channels, energy consumption for reporting local decisions and energy consumption for receiving the final cooperative decision which is given as:

$$E_{MN}^{cs}=\underset{\begin{array}{c}Channels \\ Sensing\end{array}}{\underbrace{{\text{E}}_{\text{cs}}\left(n_z,T_{cs}\right)}}+\underset{\begin{array}{c}Decision \\ Reporting\end{array}}{\underbrace{{\text{E}}_{rp}\left(B_{ld},d_{i,j}\right)}}+\underset{\begin{array}{c} Coop. Decision\\ Receiving\end{array}}{\underbrace{{\text{E}}_{rx}\left(B_{cd}\right)}}$$

(15)

Each $C{H}_j$ performs data fusion upon receiving MNs’ local decisions and then broadcasts the final cooperative decision. Energy consumption ${\text{E}}_{rx}$ for receiving $B_{ld}$-bits of each local decision is given as:

$${\text{E}}_{rx}\left(B_{ld}\right)= B_{ld}{E}_{ec}$$

(16)

Let $R_{max}$ denote the maximum radio range of clusterhead $C{H}_j$, then the consumed energy ${\text{E}}_{bd}$ for broadcasting the final cooperative decision can be expressed as:

$${\text{E}}_{bd}\left(B_{ld},d_{j,R_{max}}\right)= B_{ld}\left(E_{ec}+E_{am} d_{j,R_{max}}\right)$$

(17)

Let ${\text{E}}_{dp}$ denote energy consumption for processing a $B_{ld}$ bits packet received from each member node $M{N}_i$ for decision fusion. Energy cost for clusterhead $C{H}_j$ cooperative channel sensing is the energy consumed for sensing a set of channels, energy consumed for receiving $m_n$ member nodes’ local decisions, energy consumption for processing the decisions and energy consumption for broadcasting the final cooperative decisions which is given as:

$$E_{CH}^{cs}=\underset{\begin{array}{c}Channels \\ Sensing\end{array}}{\underbrace{{\text{E}}_{\text{cs}}\left(n_z,T_{cs}\right)}}+\underset{\begin{array}{c}Decision\\ Receiving \end{array}}{\underbrace{{\displaystyle \sum }_{e=1}^{m_n}{\text{E}}_{rx}\left(B_{ld}\right)}}+\underset{\begin{array}{c} Decision\\ Processing\end{array}}{\underbrace{{\displaystyle \sum }_{e=1}^{m_n+1}{\text{E}}_{dp}\left(B_{ld}\right)}}+\underset{\begin{array}{c}Coop.Decision\\ Broadcasting\end{array}}{\underbrace{{\text{E}}_{bd}\left(B_{ld},d_{j,R_{max}}\right)}}$$

(18)

Therefore, total energy cost for cooperative channel sensing $E_{total}^{cs}$ can be expressed as:

$$\begin{gathered} \begin{array}{c}{E}_{total}^{cs}={\displaystyle {\displaystyle \sum }_{j=1}^k}\left({\displaystyle {\displaystyle \sum }_{e=1}^{m_n}}{E}_{MN}^{cs}+E_{CH}^{cs}\right)\\ ={\displaystyle {\displaystyle \sum }_{j=1}^k}\left({\displaystyle {\displaystyle \sum }_{e=1}^{m_n}}(E_{rp}(B_{ld},d_{i,j})+2E_{rx}(B_{cd})) \\ +{\displaystyle {\displaystyle \sum }_{e=1}^{m_n+1}}(E_{cs}(n_z,T_{cs})+E_{dp}(B_{dp}))+E_{bd}(B_{cd},d_{j,R_{max}})\right)\end{array} \end{gathered}$$

(19)

3.3.4. Transceiver Unit Energy Consumption

The transceiver unit enables communication between the member nodes MNs and the clusterheads CHs as well as between the CHs and the BS. Each $M{N}_i$ transmits its reading data to any of the selected clusterheads $C{H}_j$ through the available licensed channels. Since all MNs are within the radio range of their neighboring clusterheads, adjacent MNs can send their data to the $C{H}_j$ without intermediary nodes. Energy consumption ${\text{E}}_{tM}$ for transmitting $B_{dt}$-bits packet to $C{H}_j$ over a distance $d_{i,j}$ can be expressed as:

$${\text{E}}_{tM}\left(B_{dt},d_{i,j}\right)= B_{dt}\left(E_{ec}+E_{am} d_{i,j}\right)$$

(20)

The $C{H}_j$ aggregates the data received from the MNs and then forwards the aggregated data either through some intermediate neighbouring clusterhead $C{H}_g$ or directly to the BS. Energy consumption ${\text{E}}_{rM}$ for receiving $B_{dt}$-bits packet from member node $M{N}_i$ can be expressed as:

$${\text{E}}_{rM}\left(B_{dt}\right)= B_{dt}{E}_{ec}$$

(21)

Let $C{H}_g$ denote the immediate neighbouring clusterhead through which the aggregated data packets $B_{pk}$ will be routed to the BS, $E_{mp}$ denotes the energy consumption for amplifying the signal and $d_{j,g}$ denotes the distance between clusterhead $C{H}_j$and the immediate neighbouring clusterhead $C{H}_g$ or the BS. The energy consumption ${\text{E}}_{tH}$ for transmitting the aggregated data packets $B_{pk}$ to the immediate neighboring clusterhead $C{H}_g$ or the BS over a distance $d_{j,g}$ can be expressed as:

$${\text{E}}_{tH}\left(B_{pk},d_{j,g}\right)= B_{pk}\left(E_{ec}+E_{mp} d_{j,g}\right)$$

(22)

Energy consumption for receiving the aggregated data packets $B_{pk}$ from neighbouring clusterhead $C{H}_g$ for onward transmission is given as:

$${\text{E}}_{rH}\left(B_{pk}\right)= B_{pk}{E}_{ec}$$

(23)

The source of power consumption in the RF-front end is mainly dominated by power amplifier which boosts the transmission power to a certain level depending on the type of the amplifier and the application. The power amplifier’s power consumption $P_{am}$ which is a function of transmission distance $d$ largely depends on many factors which include operating frequency, DC supply voltage, output power, hardware technology and load characteristic. The total power consumption for running the power amplifier is equal to the DC input power $P_{dc}$($P_{am}$ = $P_{dc}$) [44]. The ability of the power amplifier to convert the DC input power $P_{dc}$ into RF signal power $P_{tx}$is referred to as drain efficiency. This efficiency can be expressed as the ratio of signal power to DC power given as:

$$\eta =\frac{P_{tx}}{P_{dc}}$$

(24)

Communication over the wireless medium is susceptible to propagation impairments such as multi-path, fading and attenuation. If $P_{tx}$ denotes the RF signal power from the transmitter’s power amplifier delivered to the receiver node’s antenna, and $G$ is a parameter that defines the characteristic of the transmitting and receiving antenna, then the RF signal power $P_{rx}$ received at the receiving sensor node can be expressed as:

$$P_{rx}=\frac{P_{tx}}{\left(G d^{\beta }\right)}$$

(25)

Therefore, from Equations (24) and (25), the minimum power consumption of RF power amplifier to amplify transmission signals is given by $P_{am} =\mu /\eta$, where $\mu =P_{rx} G$is a constant given by the received RF signal power ${\text{P}}_{\text{rx}}$ and the antenna characteristic $\text{G}$ [44]. The parameter value $\text{μ}$ is a function of radio environment and can be set to a single-hop maximum transmission power value i.e., 6.3 mW instead of absolute value as in [44]. The energy consumption for data transmission comprises of energy cost for intra-cluster and inter-cluster data communication.

In intra-cluster data communication, distance between the MNs and their prospective CHs is presumably short and therefore, the channel between them follows the Friis free space path loss model with signal power attenuation of $\beta =2$ power loss [30]. Thus, energy consumption for intra-cluster communication comprises of total energy consumption of all MNs for transmitting $B_{dt}$ bits of data packet over a distance $d_{i,j}$ to their respective clusterheads $C{H}_j$ and total energy consumption for receiving the data by the clusterhead $C{H}_j$. This is given as:

$$E_{Intr}\left(i,j\right)= {\displaystyle \sum }_{\text{j}=1}^{\text{q}}{\displaystyle \sum }_{\text{i}=1}^{{\text{m}}_n}\left(\underset{\begin{array}{c} Data\\ Transmitting\end{array}}{\underbrace{{\text{E}}_{tM}\left(B_{dt},d_{i,j}\right)}}+\underset{\begin{array}{c} Data\\ Receiving\end{array}}{\underbrace{{\text{E}}_{rM}\left(B_{dt}\right)}}\right)$$

(26)

In inter-cluster data communications, the distance between CHs and BS is presumably long and therefore, the channel between them follows the Friis free space path loss model with signal power attenuation of $\beta =4$ power loss [30]. Each clusterhead $C{H}_j$ forwards its aggregated data packets $B_{pk}$ to the BS either through intermediate clusterhead $C{H}_g$ $(if d_{j,g}>R_{max,j})$ or direct to the BS without any intermediate clusterhead $C{H}_g$ $\left(if d_{j,g}\le R_{max,j}\right)$. The total energy consumption for inter-cluster data communications is the energy consumption for aggregating the received data and energy consumption for forwarding the data packet to the BS which is given as:

$$E_{int}={\displaystyle \sum }_{j=1}^q\left(\underset{\begin{array}{c}Data \\ Aggregation\end{array}}{\underbrace{E_{ap}\left(B_{ap},N_{cy}\right)}}+\underset{\begin{array}{c}Data \\ Transmitting\end{array}}{\underbrace{{\text{E}}_{tH}\left(B_{pk},d_{j,g}\right)}}\right)$$

(27)

Thus, the total energy consumption for data communications can be expressed as:

$$\begin{array}{c}{E}_{total}^{dt}=E_{Intr}\left(i,j\right)+E_{int}\\ ={\displaystyle {\displaystyle \sum }_{\text{j}=1}^q}\left({\displaystyle {\displaystyle \sum }_{\text{i}=1}^{{\text{m}}_n}}(\underset{\begin{array}{c}Data\\ Transmitting\end{array}}{\underbrace{E_{tM}\left(B_{dt},d_{ij}\right)}}+\underset{\begin{array}{c}Data\\ Receiving\end{array}}{\underbrace{{\text{E}}_{rM}\left(B_{dt}\right)}}\right)+\underset{\begin{array}{c}Data \\ Aggregation\end{array}}{\underbrace{E_{ap}\left(B_{ap},N_{cy}\right)}}+\underset{\begin{array}{c}Data \\ Transmitting\end{array}}{\underbrace{{\text{E}}_{tH}\left(B_{pk},d_{j,g}\right)}})\end{array}$$

(28)

If $m_{n,j}$ denotes the number of member nodes in cluster $c{l}_j$ and $q$ is the number of clusters in the network which is also equal to the number of clusterheads in the network, then the number of cognitive radio sensor nodes in the network $N={{\displaystyle \sum }}_{j=1}^q\left(m_{n,j}+1\right)$. Therefore, total energy consumed by member node $M{N}_i$ in cluster $c{l}_j$ is given as:

$$\begin{array}{c}{E}_{MN}(i,j)=\underset{\begin{array}{c}Events\\ Sensing\end{array}}{\underbrace{E_{ss}\left(B_{ss}\right)}}+\underset{\begin{array}{c}Data\\ Logging\end{array}}{\underbrace{E_{log}\left(B_{lg}\right)}}+\underset{\begin{array}{c}Channel\\ sensing\end{array}}{\underbrace{{\text{E}}_{cs}(n_z,T_{cs})}}+\underset{\begin{array}{c}Decision\\ Reporting\end{array}}{\underbrace{{\text{E}}_{rp}(B_{ld},d_{i,j})}}+\underset{\begin{array}{c}Coop.Decision\\ Receiving\end{array}}{\underbrace{{\text{E}}_{rx}\left(B_{ld}\right)}}+\underset{\begin{array}{c}Data\\ Transmitting\end{array}}{\underbrace{{\text{E}}_{tM}(B_{dt},d_{ij})}}\\ \end{array}=P_{ss} x T_{ss} x B_{ss}+B_{lg}(P_{rd}+ P_{wr})+{\displaystyle \sum }_{m=1}^{ n_z}(T_{cs}{\text{P}}_{ed}+{\text{E}}_{sp})+B_{ld}\left(2E_{ec}+E_{am} d_{i,j}^{\beta }\right)+B_{dt}(E_{ec}+E_{am} d_{i,j}^{\beta })$$

(29)

Similarly, total energy consumption of clusterhead $C{H}_j$ in cluster $c{l}_j$ is given as:

$$\begin{array}{c}{E}_{CH}(j)=\underset{\begin{array}{c}Events\\ Sensing\end{array}}{\underbrace{E_{ss}(B_{ss})}}+\underset{\begin{array}{c}Data\\ Logging\end{array}}{\underbrace{E_{\mathit{log}}(B_{\mathit{lg}})}}+\underset{\begin{array}{c}Channel\\ sensing\end{array}}{\underbrace{{\text{E}}_{cs}(n_z,T_{cs})}}+{\displaystyle \sum _{i=1}^{{\text{m}}_n}\left(\underset{\begin{array}{c}Decisions\\ \mathit{Receiving}\end{array}}{\underbrace{E_{rx}(B_{ld})}}+\underset{\begin{array}{c}Data\\ \mathit{Receiving}\end{array}}{\underbrace{E_{rM}(B_{dt})}}\right)}\\ +{\displaystyle \sum _{i=1}^{{\text{m}}_n+1}\left(\underset{\begin{array}{c}Decisions\\ \mathit{Processing}\end{array}}{\underbrace{E_{dp}(B_{dp})}}+\underset{\begin{array}{c}Data\\ Aggregation\end{array}}{\underbrace{E_{ap}(B_{dt},N_{cy})}}\right)+\underset{\begin{array}{c}Data\\ Transmitting\end{array}}{\underbrace{{\text{E}}_{tM}(B_{dt},d_{i,j})}}}\\ =P_{ss} x T_{ss} x B_{ss}+B_{\lg }(P_{rd}+P_{wr})+{\displaystyle \sum _{m=1}^{n_z}(T_{cs} P_{ed}+E_{sp})}+{\displaystyle \sum _{i=1}^{{\text{m}}_n}(B_{ld} E_{ec}+B_{dt} E_{ec})}\\ +{\displaystyle \sum _{i=1}^{{\text{m}}_n+1}\left(E_{dp}(B_{dp} )+B_{ap}{N}_{cy}{C}_{av}{V}_s^2+B_{ap}{V}_s\left(I_o{e}^{\frac{V_s}{n_p{V}_t}}\right)\left(\frac{N_{cy}}{f}\right)\right)}\\ +B_{cd}\left(E_{ec}+E_{am} d_{j,R_{\max }}^{\beta }\right)+B_{pk}{P}_{ec}+B_{pk}\left(E_{ec}+E_{mp} d_{j,g}^{\beta }\right)\end{array}$$

(30)

Thus, total energy consumption for the entire network is given as:

$$E_{net}={{\displaystyle \sum }}_{j=1}^q\left(E_{CH}\left(j\right)+{{\displaystyle \sum }}_{i=1}^{m_{n,j}}{E}_{MN}\left(i,j\right)\right)$$

(31)

3.4. Optimal Number of Clusters

Network clustering is a promising technique that can be employed to achieve network scalability, reliable and energy efficient communication. A near-centre member node $M{N}_i$ in a cluster consumes less energy for intra-cluster communication than a near-border member node $M{N}_g$ but both consume maximum power for inter-cluster communication when they are selected as clusterheads [17]. This means a shorter average distance between member nodes and clusterhead requires less energy for intra-cluster communication. Therefore, the number of clusters which influences member nodes distribution in each cluster and average intra-cluster distance are key elements to be considered in minimizing network energy consumption. The optimal number of clusters $q^{*}$ need to be carefully determined so that network-wide energy consumption can be minimized. The network is partitioned into $q$ number of clusters with each cluster comprises of one clusterhead $C{H}_j$ and $1-\left(\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$q$}\right.\right)$ number of member nodes $m_n\approx \raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$q$}\right.$ uniformly distributed within the radio range of the clusterhead $C{H}_j$. Let $A_{sp}^2$ denotes average spans for $x$-axis and $y$-axis of each cluster $c{l}_j$, $A_c$ denotes average cluster area and $\rho \left(x,y\right)$ denotes nodes distribution density within a cluster. The Euclidian square distance $d_{i,j}^2$ between the member node $M{N}_i$and the clusterhead $C{H}_j$ can be expressed as:

$$d_{i,j}^2=\iint \left(x^2+y^2\right)\rho \left(x,y\right)dxdy$$

(32)

If area $A_{c,i}$ of cluster $c{l}_j$ is assumed to be a two dimensional area, then the average span $A_{sp}^2=L^2/q$ and the node distribution density $\rho =q/L^2$. Thus, Equation (32) can be expressed as:

$$d_{i,j}^2=\rho {{\displaystyle \int }}_{x=0}^{L/\sqrt{q}}{{\displaystyle \int }}_{y=0}^{L/\sqrt{q}}\left(x^2+y^2\right)dxdy$$

(33)

After further simplification, the equation reduces to:

$$d_{i,j}^2 =\frac{2}{3}\frac{L^2}{q}$$

(34)

Therefore, optimal number of cluster $q^{*}$ can be derived analytically from the network wide energy consumption equation which is given as:

$$\begin{gathered} \begin{array}{c}{E}_{net}={\displaystyle \sum _{j=1}^q\left(E_{CH}(j)+{\displaystyle \sum _{i=1}^{m_{n,j}}{E}_{MN}(i,j)}\right)}\\ ={\displaystyle {\sum }_{j=1}^q\left(E_{ss}+E_{log}+{\text{E}}_{\text{cs}}+{\displaystyle {\sum }_{i=1}^{{\text{m}}_n}\left({\text{B}}_{ld}{E}_{ec}+B_{dt}{E}_{ec}\right)+{\displaystyle {\sum }_{i=1}^{{\text{m}}_n+1}\left({\text{E}}_{dp}+E_{ap}\right) \\ +B_{cd}\left(E_{ec}+E_{am} d_{j,R_{\max }}^{\beta }\right)+B_{pk}{E}_{ec}+B_{pk}\left(E_{ec}+E_{mp} d_{j,g}^4\right)}}+{\displaystyle {\sum }_{i=1}^{m_{n,j}}\left(E_{ss}+E_{log}+ \\ {\text{E}}_{\text{cs}}+B_{ld}\left(2E_{ec}+E_{am} d_{i,j}^{\beta }\right)+B_{dt}(E_{ec}+E_{am} d_{i,j}^{\beta })\right)}\right)}\end{array} \end{gathered}$$

(35)

Let $B$ stand for $B_{ld},$ $B_{dt}$, $B_{pk}$ and $\beta =2$ for intra-cluster distance while $\beta =4$ for inter-cluster distance i.e., clusterhead to BS:

$$\begin{gathered} E_{net}=\left(\left(q{E}_{ss}+q{E}_{log}+ {\text{qE}}_{\text{cs}}+2NB{E}_{ec}+N\left({\text{E}}_{dp}+E_{ap}\right)+Bq\left(3E_{ec}+\frac{2E_{am}}{3}\frac{L^2}{q}\right) \\ +\text{Bq}{E}_{mp} d_{j,g}^4+N\left(E_{ss}+E_{log}+ {\text{E}}_{\text{cs}}+B\left(3E_{ec}+\frac{4E_{am}}{3}\frac{L^2}{q}\right)\right)\right)\right) \end{gathered}$$

(36)

Therefore, the optimal number of clusters $k_{opt}$ can be determined by setting derivative of $E_{net}$ in Equation (36) with respect to $q$ to equal to zero $\frac{\partial E_{net}}{\partial q}=0$. After further derivation and simplification, the equation reduces to:

$$q^{*}=\sqrt{\frac{4\text{N}{E}_{am}{\text{A}}_N}{3(E_{pp}+2E_{ec}+{\text{E}}_{tH})}}$$

(37)

where $E_{pp}=\left(E_{ss}+E_{log}+ {\text{E}}_{\text{cs}}\right)/B$ denotes the energy costs per bit for sensing events, logging the readings data and sensing set of channels for detecting vacant channels, respectively.

3.5. Modelling of RL-Based Clustering

The problem of selecting optimal clusterhead is formulated as a Markov Decision Process (MDP) where a SU learns the energy consumption and local decision accuracy for neighboring clusterheads and then selects an optimal cluster that minimizes energy consumption and improves sensing performance. A quadruple $\left(\mathcal{S},\mathcal{T},\mathcal{A},ℛ\right)$ represents the Markov Decision Process (MDP) for selecting an optimal cluster in the network, $\mathcal{S}$ denotes set of states in the model of the operating environment $\text{s}=\left\{s_1,s_2,s_3,s_4\dots ,s_n\right\},\text{ s}\in \mathcal{S}$, $\mathcal{T}$ denotes state transition function, $\mathcal{A}$ denotes a set of actions to be executed $a=\left\{a_1,a_2,a_3\dots ,a_n\right\}$, $a\in \mathcal{A}$, while $ℛ$ denotes the state reward function $r\left(s,a\right)\in ℛ$ [19]. Each of the SUs or the agent selects an action $a_k$ in every state $s_k$ of the model as shown in Figure 5. The selected action $a_k$ leads to sensing a set of channels, reporting local decisions to a clusterhead, computing the energy consumption and evaluating the local decision accuracy for the chosen cluster. Reward $r_{k+1}$ obtained from the computed energy consumption and local decision accuracy for state $s_k$ determines the next state $s_{k+1}$ and the next action $a_{k+1}$, $k$ denotes the stage index of the process. The agent adopts an optimal policy $\pi$ that maximizes the cumulative reward obtained from a known state experience and from exploitation of unknown states to select the optimal clusterhead.

Figure 5. Model of reinforcement learning-based clustering.

Figure 5. Model of reinforcement learning-based clustering.

States: The state $s_k$ of the MDP stands for the stage at which the agent selects a cluster among the neighboring clusterheads and determines the reward for taking an action $a_k$ in the state. The set $\left|S\right|$ comprises of states equal to the number of neighbouring clusterheads plus an initialization state which initiates the state transition for selecting the clusterhead. Initialization of the clusterhead selection process begins at $k=0$, $v=0$ for $s_{\text{k}}^t=v$, where $t= \left\{0, 1, 2, \mathrm{...},\text{T}\right\}$ denotes the decision episode, $k$ denotes the stage index and $v\in \mathcal{S}$ denotes the current state number which indicates the selected clusterhead, if $v\ne 0$. Therefore, at every stage index $k\ne 0$ of the learning process, the state $s_k^t= v\in \mathcal{S}$ for selecting an action $a_k^t\in \mathcal{A}$ can be expressed as:

$$s_k^t=v . I_x\left(s\right)$$

(38)

where $I_x\left(s\right)$ is an indicator function in that:

$$I_x\left(s\right)=\{\begin{array}{c}1, if a_k^t\ne 0\\ 0, otherwise\end{array}$$

(39)

For each episode $t$ of the learning process, the agent $i$ employs a softmax action selection strategy to select an action $a_k^t$, computes the state-action reward $r_k^t\in ℛ$ for the current state $s_k^t=v$, and then determines the subsequent state $s_{k+1}^t=h\in \mathcal{S}$.

Actions: an action $a_k$in this context implies a strategic choice made by an agent for selecting a clusterhead $C{H}_j$among the neighbouring clusterheads $C{H}_j|j=1, 2, 3,\dots ,n_{hg}$. The selected action $a_k^t=j\in \mathcal{A}$ in every state $s_k^t=v\in \mathcal{S}$ is expected to maximize the current reward $r_k^t$ for updating Q-value $Q_k^t\left(s_k^t,a_k^t\right)$. If $Q_k^t=(s_0^t$,$a_0^t,s_1^t,a_1^t,\dots ,s_{k-1}^t,a_{k-1}^t, s_k^t,a_k^t)$ denotes the sequence of state-actions executed from $s_0^t$to $s_k^t$ in episodes $t= \left\{0, 1, 2, \mathrm{...},\text{ T}\right\}$ and $C{H}_{k-1}^t$ denotes the corresponding set of selected clusterheads for the state-actions then the action taken can be formulated based on a stochastic process as $a_k^t={\sigma }_k^t\left(Q_k^t\right)\in \left({\mathcal{A}}_{s_k^t}\right)$, where ${\mathcal{A}}_{s_k^t}=\mathcal{A}\backslash \{\left\{s_k^t\right\}{{\displaystyle \cup }}^{}C{H}_{k-1}^t$, denotes a set of selected actions, ${\sigma }_k^t$ denotes decision rule that maps the sequence of state-action $Q_k^t$ into a probability distribution ${\Delta }_{{\mathit{\sigma }}_k^t}\left({\mathcal{A}}_{s_k^t}\right)$ [3].

To achieve a policy $\pi$ that maximizes long-term rewards, a Boltzmann distribution-based action selection strategy known as softmax is adopted to balance between exploration of random actions and exploitation of state-actions. The strategy selects an action that returns the highest estimated reward for the state-action values based on a probability $\mathcal{p}$ determined by a positive parameter ${\tau }^t$called temperature. The probability $\mathcal{p}$ can be expressed as [27]:

$$\mathcal{p}\left(s_k^t,a_k^t=j \right)=\frac{e^{Q\left(s_k^t,a_k^t=j\right)/{\tau }^t}}{{{\displaystyle \sum }}_{\overline{h}}^{\left|{\mathcal{A}}_{s_k^t}\right|}{e}^{Q\left(s_k^t,a_k^t=\overline{h}\right)/{\tau }^t}}, j\in {\mathcal{A}}_{s_k^t}$$

(40)

Transition Probability: The transition probability $\mathcal{T}: S \times A \times \mathcal{S}\dots .S \to \left[0,1\right]$maps the state-action transit to a probability of moving from current state $s_k^t$ to next state $s_{k+1}^t$ whenever an action $a_k^t$ is executed in the state $s_k^t$. Therefore, the transition probability $P_m\left(s_k^t/s_{k+1}^t, a_k^t\right)$from current state $s_k^t=v$ to the next state $s_{k+1}^t=h$is a function of the action $a_k^t=j$ executed in the current state. However, the adopted learning algorithm which is Q-learning does not require transition probabilities.

Reward function: Reward $ℛ :S \times A \times S\dots . \mathcal{S} \to r_k^t$ is a key component of the MDP model that can be used to evaluate the state-action value for each episode and update the Q-table. It maps the state transition from state $s_k^t$ to subsequent state $s_{k+1}^t$ for the action $a_k^t$taken to an actual value reward [3]. In each episode, the agent selects an action $a_{\text{k}}^t=\text{j}$ to select clusterhead $C{H}_j$ in state $s_{\text{k}}^t=v$, computes its reward $r_{\text{k}}^t$ before moving to the next state $s_{\text{k}}^t=h$, and then restarts the state $s_k^t=0$ upon reaching the last state. The expected cumulative reward $r_{k+1}^t$can be computed based on agent’s reward for energy consumption $r{w}_{E,k+1}^t$ and reward for local decision accuracy $r{w}_{D,k+1}^t$ subject to the pairwise constraint condition $P_{w,k+1}^t$. Thus:

$$r_{k+1}^t \left(s_k^t, a_k^t\right)= \frac{1}{2}\left[ r{w}_{E,k+1}^t+r{w}_{D,k+1}^t \right]I_A\left\{P_{w,k+1}^t\ne 0\right\}$$

(41)

where $k =0, 1, 2,\dots ,k^t-1$ denotes the stage index at episode $t$, the indicator function $I_A\left\{x\right\}$can either be zero or one, depending on $x$. The pairwise constraint condition $P_{w,k+1}^t$ can be expressed as:

$$P_{w,k+1}^t= \{\begin{array}{c}1, M_{V,i}\cap H_{V,j}\ne \varnothing \\ 0, otherwise \end{array}$$

(42)

This means that the cumulative reward will be zero if the member node $M{N}_i$ and the chosen clusterhead $C{H}_j$ share no common vacant channel $M_{V,i}\cap H_{V,j}=\varnothing$. The reward for energy consumption $r{w}_{E,k+1}^t$ can be obtained from the member node’s $M{N}_i$ consumed energy ${\text{E}}_{tM}$ for transmitting $B_{dt}$bits of data packet to a neighboring clusterhead $C{H}_j$ and for performing cooperative channel sensing $E_{MN}^{cs}$. Thus, consumed energy is given as:

$$\begin{gathered} E_{TC,j}\left(B_{dt},d\right)= E_{MN}^{cs}+ {\text{E}}_{tM} \\ ={\displaystyle \sum }_{m=1}^{ n_z}\left(T_{cs}{P}_{ed}+{\text{E}}_{sp}\right)+B\left(3E_{ec}+2E_{am} d_{i,j}^{\beta }\right) \end{gathered}$$

(43)

Therefore, a reward of one will be obtained for minimum energy consumption, a reward of half will be awarded for energy consumption less than the maximum consumed energy, while the reward for maximum energy consumed is zero. Thus, the reward for energy consumption is given as:

$$r{w}_{E,k+1}^t=\{\begin{array}{c}1, E_{TC,j}=\underset{E_{TC,j}\in E_{TC}^{set}}{\min }\left(E_{TC}^{set}\right)\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right., E_{TC,j}<\underset{E_{TC,j}\in E_{TC}^{set}}{\max }\left(E_{TC}^{set}\right)\\ 0, E_{TC,j}=\underset{E_{TC,j}\in E_{TC}^{set}}{\max }\left(E_{TC}^{set}\right)\end{array}$$

(44)

State-action that leads to selection of a clusterhead that leads to low energy consumption receives a higher reward which translates into an increase in the corresponding Q-value and also its chances for likely selection in subsequent states. This means that less favorable clusters that lead to high energy consumptions are more likely to be excluded during the learning process.

Local decision on PU existence in a channel is usually prone to errors due to the channel’s propagation impairment which degrades the channel sensing performance. Therefore, the local decision accuracy determines the divergence of an individual member node’s sensing outcome compared to the cooperative sensing outcome. The reward for local decision accuracy can be obtained by comparing the local decision $L{D}_i^z\in L{D}_i$ made by member node $M{N}_i$ on PU existence on the channels with the cooperative decisions $C{D}_j^z\in C{D}_j$ which involves other member nodes MNs in the cluster $c{l}_j$. Therefore, when a set of local decisions $L{D}_i$ matches with a corresponding set of cooperative decisions $C{D}_j$, then a reward of one will be received, while local decisions $L{D}_i$ that agree with the majority of the corresponding cooperative decisions $C{D}_j$ receive half rewards, local decisions $L{D}_i$ that match the corresponding cooperative decisions $C{D}_j$ with less than half the number of the channels attract a reward of zero (i.e., do not earn a reward). Thus, the reward for local decision accuracy can be expressed as:

$$r{w}_{D,k+1}^t=\{\begin{array}{c}1, L{D}_i\cap C{D}_j= n_z \\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right., L{D}_i\cap C{D}_j\ge \left(n_z/2\right)\\ 0, L{D}_i\cap C{D}_j<\left(n_z/2\right)\end{array}$$

(45)

3.6. The EESA-RLC Algorithm

The algorithm begins with the clustering initialization phase which deals with clusterhead emergence as outlined in Table 1. All SUs (line 1 to line 20) sense a set of predefined channels (line 2), compute their clusterhead probabilities based on the number of vacant channels detected, percentage of clusterheads and their residual energy (line 3). If the clusterhead probability is greater or equal to the threshold, then a clusterhead announcement is broadcast (line 5) and the SU becomes a tentative clusterhead (line 6), otherwise, if the clusterhead probability is less than the threshold and it has heard the clusterhead announcement (line 7), then it waits for the final announcement (line 8) and the SU is a member node (line 9). Another set of SUs (line 10) perform iteration (line 11 to line 18). In each iteration, we increment the probability by a factor (line 12) and compare the result with the threshold (line 13). If it is clusterhead (line 15), it broadcasts an announcement (line 14). The Base Station selects an optimal number of clusterheads among the potential clusterheads (line 21) and broadcasts the list (line 24). The selected clusterheads broadcast advertisement packets to their neighboring SUs (line 24 to line 25).

Table 1. Initialization phase.

Algorithm 1: Cluster Head Emergence
Required: $\mathit{N}, {\mathbf{E}}_n^{\mathbf{Re}}, {\mathit{q}}^{*}, \mathit{\psi }, {\mathbf{E}}_{\mathit{n}}^{\mathbf{mx}}$, ξ
1: for $\mathit{n}\leftarrow \mathbf{1} \mathit{t}\mathit{o} \mathit{N} \mathit{d}\mathit{o}$
2:  Sense Channels $(\mathit{C}{\mathit{h}}_{\mathit{z}}|z=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,{\mathit{n}}_{\mathit{z}})$
3:  Compute ${\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}={\mathbf{E}}_{\mathit{n}}^{\mathbf{Re}}\hspace{0.17em}{\mathit{\phi }}_{\mathit{n}}\hspace{0.17em}\psi /{\mathbf{E}}_{\mathit{n}}^{\mathbf{mx}}\hspace{0.17em}{\mathit{n}}_{\mathit{z}}$
4:  if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\ge \mathit{\psi })$ then
5:    Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
6:    $\mathit{C}\mathit{H}=\mathit{C}\mathit{H}+\mathbf{1}$
7:  else if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}<\mathit{\psi })$ and Received ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$ then
8:    Wait for final clusterhead announcement
9:    $\mathit{M}\mathit{N}=\mathit{M}\mathit{N}+\mathit{1}$
10:   else
11:     for $\mathit{l}\leftarrow \mathbf{1} \mathit{t}\mathit{o} {\mathit{N}}_{\mathit{i}\mathit{r}} \mathit{d}\mathit{o}$
12:     ${\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\leftarrow {\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}+\hspace{0.17em}\mathbf{\xi }\mathit{\psi }$
13:     if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\ge \mathit{\psi })$ then
14:      Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
15:      $\mathit{C}\mathit{H}=\mathit{C}\mathit{H}+\mathbf{1}$
16:     Break
17:     end if
18:     end for
19:   end if
20: end for
21: BS Selects ${\mathit{q}}^{*}$ from $\mathit{C}{\mathit{H}}_{\mathit{n}}|n=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,\mathit{C}\mathit{H}$
22: Broadcasts $\mathit{H}=\left\{\mathit{C}{\mathit{H}}_{\mathit{n}}\right|\mathit{n}=\mathbf{1},\mathbf{2},\mathbf{3},\dots , {\mathit{q}}^{*}\}$
23: for $\mathit{j}=\mathbf{1} \mathit{t}\mathit{o} {\mathit{q}}^{*}\hspace{0.17em}\mathit{d}\mathit{o}$
24:   Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
25: end for

Table 1. Initialization phase.

Algorithm 1: Cluster Head Emergence
Required: $\mathit{N}, {\mathbf{E}}_n^{\mathbf{Re}}, {\mathit{q}}^{*}, \mathit{\psi }, {\mathbf{E}}_{\mathit{n}}^{\mathbf{mx}}$, ξ
1: for $\mathit{n}\leftarrow \mathbf{1} \mathit{t}\mathit{o} \mathit{N} \mathit{d}\mathit{o}$
2:  Sense Channels $(\mathit{C}{\mathit{h}}_{\mathit{z}}|z=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,{\mathit{n}}_{\mathit{z}})$
3:  Compute ${\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}={\mathbf{E}}_{\mathit{n}}^{\mathbf{Re}}\hspace{0.17em}{\mathit{\phi }}_{\mathit{n}}\hspace{0.17em}\psi /{\mathbf{E}}_{\mathit{n}}^{\mathbf{mx}}\hspace{0.17em}{\mathit{n}}_{\mathit{z}}$
4:  if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\ge \mathit{\psi })$ then
5:    Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
6:    $\mathit{C}\mathit{H}=\mathit{C}\mathit{H}+\mathbf{1}$
7:  else if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}<\mathit{\psi })$ and Received ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$ then
8:    Wait for final clusterhead announcement
9:    $\mathit{M}\mathit{N}=\mathit{M}\mathit{N}+\mathit{1}$
10:   else
11:     for $\mathit{l}\leftarrow \mathbf{1} \mathit{t}\mathit{o} {\mathit{N}}_{\mathit{i}\mathit{r}} \mathit{d}\mathit{o}$
12:     ${\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\leftarrow {\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}+\hspace{0.17em}\mathbf{\xi }\mathit{\psi }$
13:     if $({\mathit{p}}_{\mathit{n}}^{\mathit{c}\mathit{h}}\ge \mathit{\psi })$ then
14:      Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
15:      $\mathit{C}\mathit{H}=\mathit{C}\mathit{H}+\mathbf{1}$
16:     Break
17:     end if
18:     end for
19:   end if
20: end for
21: BS Selects ${\mathit{q}}^{*}$ from $\mathit{C}{\mathit{H}}_{\mathit{n}}|n=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,\mathit{C}\mathit{H}$
22: Broadcasts $\mathit{H}=\left\{\mathit{C}{\mathit{H}}_{\mathit{n}}\right|\mathit{n}=\mathbf{1},\mathbf{2},\mathbf{3},\dots , {\mathit{q}}^{*}\}$
23: for $\mathit{j}=\mathbf{1} \mathit{t}\mathit{o} {\mathit{q}}^{*}\hspace{0.17em}\mathit{d}\mathit{o}$
24:   Broadcast ${\mathit{A}}_{\mathit{p}\mathit{k}\mathit{t}}$
25: end for

The RL process shown in Table 2 begins immediately after the emergence of clusterheads by setting all state-action values $Q\left(\left|S\right|,\left|A\right|\right)$ array and action to zero (line 2) and then carrying out a number of iterations (line 3 to line 30) up to maximum episodes $E_{ps}$ specified in the inputs (line 1). The state transition follows a sequential order (line 4) and is re-started upon reaching the number of clusters (line 28) i.e., number of elements in $H_n$ (line 1). The softmax action selection strategy is employed (line 5) in each episode to select a clusterhead among the prospective clusterheads (line 7). This leads to sensing a set of channels $\left|S_{ch}\right|$ (line 8) and computing the energy consumptions (line 9) and then obtaining the associated rewards (line 11 to line 25). The state-action Q-values (line 26) are updated by the cumulative reward (line 10) and the final Q-matrix is determined after the last episode. The state-action that returns the maximum total cumulative reward value (line 31) denotes the optimal policy of selecting a clusterhead and the optimal cluster $C{H}_j^{*}$ is the index of the optimal policy which indicates the cluster (line 32).

Table 2. Reinforcement learning clustering.

Algorithm 2: RL Clustering
1: Input: $\left|\mathit{S}\right|,\hspace{0.17em}\left|\mathit{A}\right|,\hspace{0.17em}\left|{\mathit{S}}_{\mathit{c}\mathit{h}}\right|,\hspace{0.17em}{\mathit{E}}_{\mathit{p}\mathit{s}},\hspace{0.17em}\left|{\mathit{H}}_{\mathit{n}}\right|,\hspace{0.17em}{\mathit{n}}_{\mathit{h}\mathit{g}}$
2: Initialize: $\mathit{Q}(|\mathit{S}|,\hspace{0.17em}|\mathit{A}|)\leftarrow 0,\hspace{0.17em}\mathit{k}\leftarrow 0,\hspace{0.17em}{\mathit{a}}_{\mathit{k}}^{\mathit{t}}\leftarrow 0$
3: for $\mathit{t}\leftarrow \mathbf{1} \mathit{t}\mathit{o} {\mathit{E}}_{\mathit{p}\mathit{s}} \mathit{d}\mathit{o}$
4:  $\mathit{k}\leftarrow \mathit{k}+\mathbf{1}$
5:  ${\mathit{a}}_{\mathit{k}}^{\mathit{t}}\leftarrow \mathit{actionStrategy}({\mathit{s}}_{\mathit{k}}^{\mathit{t}},\mathit{Q})$
6:   if ${\mathit{a}}_{\mathit{k}}^{\mathit{t}}\ne \mathbf{0} \text{and} \mathit{k}\le {\mathit{n}}_{\mathit{h}\mathit{g}}$ then
7:    Select clusterhead $(\mathit{C}{\mathit{H}}_{\mathit{j}},{\mathit{a}}_{\mathit{k}}^{\mathit{t}}=\mathit{j})$
8:    Sense Channel $(|{\mathit{S}}_{\mathit{c}\mathit{h}}|)$
9:    Compute ${\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}}=\left\{{\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}\right|j=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,{\mathit{n}}_{\mathit{h}\mathit{g}}\}\hspace{0.17em}\text{for}\hspace{0.17em}|{\mathit{H}}_{\mathit{n}}|$
10:     Compute reward ${\mathit{r}}_{\mathit{k}+\mathbf{1}}^t\leftarrow \mathit{Average}(\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}},\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}})$
11:     if ${\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}=\mathbf{min}\hspace{0.17em}({\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}})$ then
12:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{1}$
13:     else if ${\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}<\mathbf{max}\hspace{0.17em}({\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}})$ then
14:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0.5}$
15:     else
16:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$
17:     end if
18:     Compare decision $(\mathit{L}{\mathit{\text{D}}}_i,\mathit{C}{\mathit{D}}_{\mathit{j}})$
19:     if $\mathit{L}{\mathit{D}}_{\mathit{i}}\cap \mathit{C}{\mathit{D}}_{\mathit{j}}=n_{\mathit{z}}$ then
20:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^t\leftarrow \mathbf{1}$
21:     else if $\mathit{L}{\mathit{D}}_{\mathit{i}}\cap \mathit{C}{\mathit{D}}_{\mathit{j}}\ge ({\mathit{n}}_{\mathit{z}}/\mathbf{2})$ then
22:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$.
23:     else
24:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$
25:     end if
26:     update ${\mathit{Q}}_{\mathit{k}}^{\mathit{t}}\leftarrow {\mathit{Q}}_{\mathit{k}}^{\mathit{t}}+\mathit{\alpha }[{\mathit{r}}_{\mathit{k}+\mathbf{1}}^{\mathit{t}}+\mathit{\gamma }\mathit{m}\mathit{a}{\mathit{x}}_{\mathit{a}}({\mathit{Q}}_{\mathit{k}+\mathbf{1}}^{\mathit{t}})-{\mathit{Q}}_{\mathit{k}}^{\mathit{t}}]$
27:    else
28:     $\mathit{k}=\mathbf{0}$
29:    end if
30: end for
31: $[{\mathit{H}}_{\mathit{o}\mathit{p}\mathit{t},}\mathit{I}]\leftarrow \underset{{\mathit{s}}_{\mathit{k}}^{\mathit{t}}\in \mathit{S},{\mathit{a}}_k^{\mathit{t}}\in \mathcal{A}}{\mathbf{max}}\left|{\displaystyle {\sum }_{\mathit{t}=\mathbf{1}}^{{\mathit{E}}_{\mathit{p}\mathit{s}}}{\mathit{r}}_{\mathit{k}}^{\mathit{t}}\left({\mathit{S}}_{\mathit{k}}^{\mathit{t}},{\mathit{a}}_{\mathit{k}}^{\mathit{t}}\right)}\right|$
32: $\mathit{C}{\mathit{H}}_{\mathit{j}}^{*}\leftarrow \left|\mathit{I}\right|$

Table 2. Reinforcement learning clustering.

Algorithm 2: RL Clustering
1: Input: $\left|\mathit{S}\right|,\hspace{0.17em}\left|\mathit{A}\right|,\hspace{0.17em}\left|{\mathit{S}}_{\mathit{c}\mathit{h}}\right|,\hspace{0.17em}{\mathit{E}}_{\mathit{p}\mathit{s}},\hspace{0.17em}\left|{\mathit{H}}_{\mathit{n}}\right|,\hspace{0.17em}{\mathit{n}}_{\mathit{h}\mathit{g}}$
2: Initialize: $\mathit{Q}(|\mathit{S}|,\hspace{0.17em}|\mathit{A}|)\leftarrow 0,\hspace{0.17em}\mathit{k}\leftarrow 0,\hspace{0.17em}{\mathit{a}}_{\mathit{k}}^{\mathit{t}}\leftarrow 0$
3: for $\mathit{t}\leftarrow \mathbf{1} \mathit{t}\mathit{o} {\mathit{E}}_{\mathit{p}\mathit{s}} \mathit{d}\mathit{o}$
4:  $\mathit{k}\leftarrow \mathit{k}+\mathbf{1}$
5:  ${\mathit{a}}_{\mathit{k}}^{\mathit{t}}\leftarrow \mathit{actionStrategy}({\mathit{s}}_{\mathit{k}}^{\mathit{t}},\mathit{Q})$
6:   if ${\mathit{a}}_{\mathit{k}}^{\mathit{t}}\ne \mathbf{0} \text{and} \mathit{k}\le {\mathit{n}}_{\mathit{h}\mathit{g}}$ then
7:    Select clusterhead $(\mathit{C}{\mathit{H}}_{\mathit{j}},{\mathit{a}}_{\mathit{k}}^{\mathit{t}}=\mathit{j})$
8:    Sense Channel $(|{\mathit{S}}_{\mathit{c}\mathit{h}}|)$
9:    Compute ${\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}}=\left\{{\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}\right|j=\mathbf{1},\mathbf{2},\mathbf{3},\dots ,{\mathit{n}}_{\mathit{h}\mathit{g}}\}\hspace{0.17em}\text{for}\hspace{0.17em}|{\mathit{H}}_{\mathit{n}}|$
10:     Compute reward ${\mathit{r}}_{\mathit{k}+\mathbf{1}}^t\leftarrow \mathit{Average}(\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}},\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}})$
11:     if ${\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}=\mathbf{min}\hspace{0.17em}({\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}})$ then
12:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{1}$
13:     else if ${\mathit{E}}_{\mathit{T}\mathit{C},\mathit{j}}<\mathbf{max}\hspace{0.17em}({\mathit{E}}_{\mathit{T}\mathit{C}}^{\mathit{s}\mathit{e}\mathit{t}})$ then
14:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0.5}$
15:     else
16:      $\mathit{r}{\mathit{w}}_{\mathit{E},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$
17:     end if
18:     Compare decision $(\mathit{L}{\mathit{\text{D}}}_i,\mathit{C}{\mathit{D}}_{\mathit{j}})$
19:     if $\mathit{L}{\mathit{D}}_{\mathit{i}}\cap \mathit{C}{\mathit{D}}_{\mathit{j}}=n_{\mathit{z}}$ then
20:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^t\leftarrow \mathbf{1}$
21:     else if $\mathit{L}{\mathit{D}}_{\mathit{i}}\cap \mathit{C}{\mathit{D}}_{\mathit{j}}\ge ({\mathit{n}}_{\mathit{z}}/\mathbf{2})$ then
22:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$.
23:     else
24:      $\mathit{r}{\mathit{w}}_{\mathit{D},\mathit{k}+\mathbf{1}}^{\mathit{t}}\leftarrow \mathbf{0}$
25:     end if
26:     update ${\mathit{Q}}_{\mathit{k}}^{\mathit{t}}\leftarrow {\mathit{Q}}_{\mathit{k}}^{\mathit{t}}+\mathit{\alpha }[{\mathit{r}}_{\mathit{k}+\mathbf{1}}^{\mathit{t}}+\mathit{\gamma }\mathit{m}\mathit{a}{\mathit{x}}_{\mathit{a}}({\mathit{Q}}_{\mathit{k}+\mathbf{1}}^{\mathit{t}})-{\mathit{Q}}_{\mathit{k}}^{\mathit{t}}]$
27:    else
28:     $\mathit{k}=\mathbf{0}$
29:    end if
30: end for
31: $[{\mathit{H}}_{\mathit{o}\mathit{p}\mathit{t},}\mathit{I}]\leftarrow \underset{{\mathit{s}}_{\mathit{k}}^{\mathit{t}}\in \mathit{S},{\mathit{a}}_k^{\mathit{t}}\in \mathcal{A}}{\mathbf{max}}\left|{\displaystyle {\sum }_{\mathit{t}=\mathbf{1}}^{{\mathit{E}}_{\mathit{p}\mathit{s}}}{\mathit{r}}_{\mathit{k}}^{\mathit{t}}\left({\mathit{S}}_{\mathit{k}}^{\mathit{t}},{\mathit{a}}_{\mathit{k}}^{\mathit{t}}\right)}\right|$
32: $\mathit{C}{\mathit{H}}_{\mathit{j}}^{*}\leftarrow \left|\mathit{I}\right|$

4. Performance Analysis and Evaluation

In this section, we analyze optimality of the EESA-RLC algorithm and evaluate its performance in terms of convergence, complexity and adaptability to model a free dynamic environment in achieving an optimal policy ${\pi }_k^{*}$ that minimizes energy consumption while enhancing vacant channel detection. The optimal policy ${\pi }_k^{*}$ maximizes the cumulative reward $r_k^t$ to achieve the optimal cluster $c{l}_j^{*}$ by choosing an optimal clusterhead $C{H}_j^{*}$ for the agent.

In the simulation, we assume a moderate scale network consisting of 250 SUs uniformly distributed in a 90 m × 90 m square area and five PUs randomly deployed in the network. Each PU can operate on one channel such that SUs can only access unused licensed channels for data transmission. Each channel might be free or occupied at any time depending on PU activity. We also consider low power wireless sensor nodes in computing the energy dissipation for spectrum sensing, reporting to clusterheads and data transmission. All SUs are homogenous and uniformly distributed in the network area, therefore in our simulation and mathematical modeling, all radio parameter and energy dissipation values for processing the received signal samples, tuning the detector’s circuit to channel’s bandwidth and running the electronics circuit are the same. The parameter values used for both analysis and simulation as indicated in Table 3 are either computed based on parameter values obtained from the indicated references or originated from the sources indicated in situ.

The Q-learning and SARSA algorithms were implemented in MATLAB to evaluate the performance of the EESA-RLC algorithm for $E_{ps}=5000$ episodes [19]. We set the step size for the exploration of state-action pairs and for learning rate update to ${\alpha }^k=a/\left(b+k\right)$, while the discount factor is set to $\gamma =0.9$. The discount factor determines the level priority given to future rewards. A factor of zero makes an agent consider only immediate rewards [48].

After extensive state-actions exploration and clusters exploitation during the learning phase, the EESA-RLC algorithm partitions the network into seven clusters $\left\{c{l}_1,c{l}_2,c{l}_3,c{l}_4,c{l}_5,c{l}_6,c{l}_7\right\}$ which translate into seven different clusterheads $\left\{C{H}_1,C{H}_2,C{H}_3,C{H}_4,C{H}_5,C{H}_6,C{H}_7\right\}$ as shown in Figure 6. The network consists of 250 uniformly distributed SUs and five PUs randomly deployed in the network.

The SUs form seven clusterheads and 243 member nodes. During the learning phase, each member node considers only clusterheads that are within its radio range instead of all seven clusters and chooses the optimal cluster among its neighborhood clusters. The seven clusters which are the optimal number for the network as determined through simulation as shown in Figure 6, are indicated by dashed circles with the corresponding common available channels inscribed in the circle.

To examine the effect of cluster size on the network energy consumption, we determined the energy consumed by both member nodes and their respective clusterheads for different cluster sizes through simulation, as shown in Figure 7. For a fixed number of nodes, when the number of clusters is too small (e.g., 3), each cluster would have a large number of member nodes that communicate with high transmission power. This increases the network energy consumption due to long intra-cluster distance communication between the member nodes and their clusterheads.

Table 3. Value of parameters used in the simulation.

Parameter	Description	Value
$N$	Number of SUs cognitive radio sensor nodes	250
$n_z$	Number of licensed channels	5
$ℳ$	Number of primary users	5
BS	Base Station Coordinate	75,75
$d_{i,j}$	Maximum distances for intra-cluster transmission	$\le 45$ m [45]
$d_{j,g}$	Maximum distance for inter-clusters transmission	$\le 150$ m [45]
$E_{psd}$	Number of episodes	5000
${\alpha }^k$	Step Size for learning rate	$a/\left(b+k\right)$, a = 1, b = 10;
$\gamma$	Discounted factor	0.9
$pkt$	Packet Size	32 byte
${\text{E}}_n^{\text{mx}}$	Initial Energy	1250 mJ
$B_W$	Channel Bandwidth	$C{h}_1:650$ MHz $C{h}_2:600$ MHz $C{h}_3:200$ MHz $C{h}_4:750$ MHz $C{h}_5:50$ MHz
$\epsilon$	Tradeoff between exploration and exploitation	0.7
$E_{sp}$	Energy dissipation: signal samples processing	150 nJ/bit [49]
$E_{ap}$	Energy dissipation: data aggregation	5 nJ/bit [45]
$E_{ec}$	Electronics dissipation: electronics circuit	50 nJ/bit [50]
$E_{ss}$	Energy dissipation: event sensing	33.75 μJ/bit [45]
$E_{log}$	Energy dissipation: data logging	81.4 μJ/bit[45]
$E_{am}$	Energy dissipation: amplifier, intra-cluster	77 pJ/bit/m2 [45]
$E_{mp}$	Energy dissipation: amplifier, inter-cluster	0.002 pJ/bit/m4 [45]
$P_{ed}$	Power consumption: tuning detector’s circuit	40 mW [49]
$E_{dp}$	Energy dissipation: data fusion	0.187 μJ/bit [49]
$\eta$	Power amplifier drain efficiency	92.4%[51]

Table 3. Value of parameters used in the simulation.

Parameter	Description	Value
$N$	Number of SUs cognitive radio sensor nodes	250
$n_z$	Number of licensed channels	5
$ℳ$	Number of primary users	5
BS	Base Station Coordinate	75,75
$d_{i,j}$	Maximum distances for intra-cluster transmission	$\le 45$ m [45]
$d_{j,g}$	Maximum distance for inter-clusters transmission	$\le 150$ m [45]
$E_{psd}$	Number of episodes	5000
${\alpha }^k$	Step Size for learning rate	$a/\left(b+k\right)$, a = 1, b = 10;
$\gamma$	Discounted factor	0.9
$pkt$	Packet Size	32 byte
${\text{E}}_n^{\text{mx}}$	Initial Energy	1250 mJ
$B_W$	Channel Bandwidth	$C{h}_1:650$ MHz $C{h}_2:600$ MHz $C{h}_3:200$ MHz $C{h}_4:750$ MHz $C{h}_5:50$ MHz
$\epsilon$	Tradeoff between exploration and exploitation	0.7
$E_{sp}$	Energy dissipation: signal samples processing	150 nJ/bit [49]
$E_{ap}$	Energy dissipation: data aggregation	5 nJ/bit [45]
$E_{ec}$	Electronics dissipation: electronics circuit	50 nJ/bit [50]
$E_{ss}$	Energy dissipation: event sensing	33.75 μJ/bit [45]
$E_{log}$	Energy dissipation: data logging	81.4 μJ/bit[45]
$E_{am}$	Energy dissipation: amplifier, intra-cluster	77 pJ/bit/m2 [45]
$E_{mp}$	Energy dissipation: amplifier, inter-cluster	0.002 pJ/bit/m4 [45]
$P_{ed}$	Power consumption: tuning detector’s circuit	40 mW [49]
$E_{dp}$	Energy dissipation: data fusion	0.187 μJ/bit [49]
$\eta$	Power amplifier drain efficiency	92.4%[51]

Figure 6. Pairwise spectrum-aware clustering result.

Figure 6. Pairwise spectrum-aware clustering result.

Figure 7. Energy consumption for clusters size.

Figure 7. Energy consumption for clusters size.

On the other hand, for large number of clusters (e.g., 15), the network energy consumption is relatively small, but inter-cluster communication consumes relatively high energy because of the excessive number of clusterheads which in reality consume much more energy than member nodes. Therefore, network energy consumption can be minimized by determining the optimal number of clusters that balances energy consumption for inter-cluster and intra-cluster communications. The result shows that at the minimum network energy consumption, the optimal number of clusters is seven.

4.1. Optimality of EESA-RLC Algorithm

The EESA-RLC algorithm allows SUs to learn and adapt to the dynamic environment to achieve an optimal solution through an optimal policy. The optimal solution means the optimal clusterhead $C{H}_j^{*}\in H_n$ selected by the cluster member node $M{N}_i$through the optimal policy ${\pi }^{*}$that maximizes cumulative reward $r_k^t$. The necessary conditions required to achieve the optimal solution is presented in the following proposition and proof.

Proposition 1. Consider a given set of neighboring clusterheads $H_n=\left\{C{H}_j|j=1, 2, 3,\dots \dots n_{hg}\right\}$ located at different distances $di{s}_n=\left\{d_{ij}|j=1, 2, 3,\dots ,n_{hg}\right\}$ from a member node $M{N}_i$. The corresponding set of energy consumptions $E_{TC}^{set}\left(B_{dt},d\right)=\left\{E_{TC,j}\right|j=1, 2, 3,\dots ,n_{hg}\}$ for the member node $M{N}_i$ to transmit its reading data to each of the clusterhead $C{H}_j|j=1,2,3\dots .n_{hg}$ is a function of distance $d_{ij}$ to the clusterhead. Each clusterhead$C{H}_j$ and member node $M{N}_i$ sense set of channels $S_{ch}=\left\{C{h}_z|z=1, 2, 3,\dots \dots n_z\right\}$ and detect $M_{V,i}=\left\{C{h}_z|z=1, 2, 3,\dots \dots n_{mv}\right\}$ and $H_{V,j}=\left\{C{h}_z|z=1, 2, 3,\dots \dots n_{hv}\right\}$ set of vacant channels respectively. If a selected clusterhead$C{H}_j$ with $\underset{d_{ij}\in di{s}_n}{\mathbf{min}}{d}_{ij}$ and$\underset{M_{V,i},H_{V,j}ϵS_{ch}}{\mathbf{max}}\left(M_{V,i}\cap H_{V,j}\right)$ maximizes cumulative reward $R_k=\frac{1}{2}\left[ r{w}_{E,k+1}^t+r{w}_{D,k+1}^t \right]$ such that $R_k=\underset{s_k^t\in \mathit{S}, a_k^t\in \mathcal{A}}{\mathbf{max}}\left|{\displaystyle \sum }_{\mathit{t}=\mathbf{1}}^{{\mathit{E}}_{\mathit{p}\mathit{s}}}{r}_k^t \left(s_k^t, a_k^t\right)\right|$, then there exist an optimal action selection ${{\pi }^{*}}_k=\left(a_1^{*}, a_2^{*},\dots \dots .a_k^{*}\right)$ that selects optimal clusterhead $C{H}_j^{*}$, where $a_k^{*}=\underset{{\mathit{a}}_{\mathit{k}}\in \mathcal{A}}{argmax} r_k\left(s_k, a_k\right)$.

Proof. In every state $s_k$, a member node $M{N}_i$ selects a clusterhead $C{H}_j\in H_n$ among the set of clusterheads to determine a set of common vacant channels $M_{V,i}\cap H_{V,j}$ and energy consumption $E_{TC,j}\left(B_{dt},d\right)\in E_{TC}^{set}\left(B_{dt},d\right)$ for transmitting data to the clusterhead over distance $d_{ij}$ through the vacant channels. The selected clusterhead $C{H}_j$ with minimum distance $\underset{d_{ij}\in di{s}_n}{\mathbf{min}}{d}_{ij}$ must minimize energy consumption $E_{TC,j}\in E_{TC}^{set}\left(B_{dt},d\right)$, maximize vacant channels detection $\underset{M_{V,i},H_{V,j}ϵS_{ch}}{\mathbf{max}}\left(M_{V,i}\cap H_{V,j}\right)$ and satisfy the pairwise constraint condition $P_{w,k+1}^t$ such that $M_{V,i}\cap H_{V,j}\ne \varnothing$ and obtain the highest reward value for the energy consumption reward $r{w}_{E,k+1}^t$ and highest reward value for the local decision accuracy reward $r{w}_{D,k+1}^t$ to achieve the maximum cumulative reward value $R_k=\underset{s_k^t\in \mathit{S}, a_k^t\in \mathcal{A}}{\mathbf{max}}\left|{\displaystyle \sum }_{\mathit{t}=\mathbf{1}}^{{\mathit{E}}_{\mathit{p}\mathit{s}}}{r}_k^t \left(s_k^t, a_k^t\right)\right|$. If $j$ denotes the index of the selected clusterhead $C{H}_j\in H_n$ in states $s_{1}to s_k$ that maximizes the cumulative reward $R_k$, then for each state $s_k, \text{j}=\underset{a{ }_k\in \mathcal{A}}{argmax} r_k\left(s_k,a_k\right)$ which denotes the optimal action selection policy ${\pi }^{*}=a_k^{*}$and the optimal clusterhead $C{H}_j^{*}\in H_n$.

4.2. Convergence and Computational Complexity of the EESA-RLC Algorithm

The algorithm converges to an optimal solution after adequate exploration and exploitation of state-actions pairs. The algorithm’s convergence period is a function of some parameter values which include action selection strategy, discount factor and learning rate. We simulate the Q-learning and SARSA RL algorithms to examine the convergence of the algorithms over $E_{psd}=5000$ episodes and evaluate their performance as shown in Figure 8. The result indicates that both the learning algorithms converged to the optimal solution but after different numbers of episodes. The SARSA learning algorithm converged to the optimal value after $E_{psd}=3020$ episodes and achieved a maximum average expected cumulative reward value of $r_{Av}^t=0.52$ which is much higher than that of the Q-learning algorithm. On the other hand, the Q-learning algorithm converged to an optimal solution at $E_{psd}=2020$ which is much lower than SARSA and achieved a maximum expected cumulative reward value of $r_{Av}^t=0.44$. This suggests that the Q-learning algorithm converges to optimal solution in a relatively short learning period because of its reliance on an action selection strategy rather than cluster exploration to update its estimated optimal policy. In contrast, cluster explorations while updating the Q-value slows the convergence of the SARSA algorithm due to the extension of its learning period, but this of course yields a better expected accumulative reward. It can be concluded that learning period has a significant effect on the convergence of the algorithms, increasing as the number of episodes increases which in turn decreases the learning rate and therefore, the algorithm converges slower.

Figure 8. Expected cumulative rewards and algorithms convergence.

Figure 8. Expected cumulative rewards and algorithms convergence.

Optimal clustering can be achieved through many techniques. While some techniques employ computationally infeasible approaches such as exhaustive search techniques which try all possible options for efficient clustering and select the best option among them, while others employ less computationally complex approaches based on heuristic techniques such as hierarchical, k-means, c-means and fuzzy clustering, etc. For example, the GWSA approach performs a high number of iterations to merge the nearest clusters in the nodes’ proximity matrix till an optimal solution is achieved [17]. However, its overall computational complexity $O\left(N^2 logN\right)$ is extremely high and increases proportionally with increase in size of the number of cognitive radio sensor nodes which make it impractical to implement in a large scale network. Distributed Group Wise Spectrum Aware (DGWSA) offers relatively low complexity as shown in Figure 9 [18]. This approach merges multiple neighboring cluster pairs at every iteration instead of using a proximity matrix and its complexity largely depends on the size of neighboring clusters instead of network size. Therefore, we compare the computational complexity of our algorithm with that of the GWSA and DGWSA algorithms by increasing the size of the cognitive radio sensor nodes as shown in Figure 9. The result shows that the GWSA algorithm offers high computational complexity, which increases with size of the network, while our algorithm and the DGWSA one achieve much lower complexities that are largely influenced by the number of neighboring clusters.

We employed Q-learning to evaluate the performance of our algorithm in terms of learning and adaptability to a dynamic environment in achieving an optimal solution. Our choice for Q-learning is due to its faster convergece which requires a shorter learning period. We examined the performance of a single MN that learns and determines cumulative average rewards for each selected clusterhead in a total number episodes of $E_{psd}=5000$, as shown in Figure 10. We also examined the influence of the two metric functions, viz energy consumption and local decision accuracy, on the optimal cluster selection as shown in Figure 11.

Figure 9. Comparison of computational complexity between EESA-RLC and other approaches.

Figure 9. Comparison of computational complexity between EESA-RLC and other approaches.

Figure 10. Average cumulative rewards for clusters.

Figure 10. Average cumulative rewards for clusters.

Figure 11. Average rewards for energy consumption and local decision.

Figure 11. Average rewards for energy consumption and local decision.

The cumulative average reward represents the average reward obtained for energy consumption and local decision accuracy during the learning process. Figure 10 shows cumulative average for the three clusterheads $C{H}_1,C{H}_2,C{H}_3$ that are within the radio range of the MN without the remaining four clusterheads $C{H}_4,C{H}_5,C{H}_6,C{H}_7$ that have no direct link with the MN. The result indicates that clusterhead $C{H}_1$ attracts the highest cumulative average reward of $r_{Av}^t=0.65$, followed by clusterhead $C{H}_2$ which attracts cumulative average reward of $r_{Av}^t=0.4$, while clusterhead $C{H}_3$ receives the lowest cumulative average reward of $r_{Av}^t=0.12$. This indicates that the agent learns and adapts to the environment through exploration of the neighbouring clusterheads and exploitation of actions that attract favourable rewards to return clusterhead $C{H}_1$ as the optimal clusterhead.

Figure 11 shows the effect of energy dissipation and local decision accuracy on selecting a clusterhead during the learning phase and choosing the optimal clusterhead. The agent learns the energy and cooperative sensing costs for each of the neighboring clusterhead and then selects the most favorable clusterhead that satisfies the pairwise constraints and minimum energy dissipation requirements as the optimal clusterhead. The local decision accuracy indicates the success of individual MN’s local decision about channels occupancy in respect to cooperative decision. In this context, local decisions are considered to be accurate when local decisions about a set of channels agree with the cooperative decisions irrespective whether the channels are available or not. Therefore, it can be deduced from the result, all the three clusterheads $C{H}_1$,$C{H}_2$ and$C{H}_3$ have satisfied the pairwise constraint which requires at least one common available channel between the MN and the clusterhead but only clusterhead $C{H}_1$ satisfied the minimum energy consumption requirement with the highest reward value $r{w}_E^t=0.98$. This suggests that significant amount of energy can be saved when the MN selects $C{H}_1$ as the optimal cluster as compared to $C{H}_3$ which attracts zero reward $r{w}_E^t=0$ for energy consumption. Therefore, it is extremely important to consider not only cooperative sensing success, but also energy consumption when choosing the optimal clusters. To evaluate the performance of our algorithm, we used GWSA [17] clustering as the benchmark for comparison, since it also considers spectrum-aware constraints in the network clustering and converges to optimal clusters. We first implemented the algorithms and obtained the optimal clustering through simulation, and then compared the performance of GWSA with our algorithm in terms of network energy minimization and spectrum sensing enhancement. Based on member nodes’ distances to their respective clusterheads in each cluster and clusterheads’ distances to the BS obtained from each of the clustering scheme, we determined Sum of Square Error (SSE) for the network, and computed network energy consumption and determined probability of detection for the two schemes as shown in Figure 12, Figure 13 and Figure 14 respectively.

Figure 12 compares the average SSE for different numbers of member nodes for the two clustering schemes. The SSE is a key component for determining the performance of clustering schemes in terms of network energy efficiency. Less SSE translates into minimum network energy consumption, which means more energy efficiency can be achieved with the clustering schemes that has less SSE. The result shows that SSE increases along with increase in number of member nodes. It is observed that our clustering scheme achieves much lower SSE than the GWSA. For example SSE for $200$ member nodes is approximately $100$ which is 50% less than that of GWSA for the same size of member nodes. This suggests that our approach is more energy efficient.

Figure 12. Average SSE for CRSN size.

Figure 12. Average SSE for CRSN size.

Figure 13. Average energy consumption for clusters.

Figure 13. Average energy consumption for clusters.

Figure 14. Average cooperative probability of detection for various CRSN size.

Figure 14. Average cooperative probability of detection for various CRSN size.

Figure 13 further reveals the performance improvement of our approach over the GWSA in terms energy efficiency. The result indicates that average energy dissipation decreases with increase in size of clusters. It is evident from the result that our approach achieved the least average energy dissipation compared with the GWSA approach. For example, the average energy dissipation for a five-cluster size of our approach is about 60 J, which is 8.4% lower than that of GWSA approach for the same cluster size. Furthermore, the total network energy dissipation based on our clustering scheme is 417.3 J, while that of GWSA is 458.8 J. This implies that an energy savings of 9% can be achieved with our approach. This shows that the RL-based approach learns energy dissipation for each neighboring cluster through exploration of the clusters and chooses the most favorable cluster that attracts low dissipation through exploitation.

Figure 14 compares the average probability of detection Qd for the two clustering schemes with different numbers of cognitive radio sensor nodes. The result indicates that our approach performs much better than the GWSA clustering scheme in terms PU detection. This is because the GWSA approach does not incorporate cooperative spectrum sensing which is vital for improving PU detection. It is observed that the average cooperative probability of detection Qd of our approach rapidly reaches a relatively high value that satisfies the required detection accuracy of ${\overline{\mathit{Q}}}_{\mathit{d}}\ge \mathbf{0.9}$ at the initial stage and then increases slowly with increasing number of cognitive radio sensor nodes. This suggests that multi-user sensing diversity exploration is crucial for enhancing PU detection and discovering of more spectrum opportunity.

5. Conclusions

In this paper, we propose a novel spectrum-aware clustering algorithm based on reinforcement learning to minimize network energy consumption and enhance channel sensing in cognitive radio sensor networks. We first modelled the network energy consumptions in terms of cooperative channel sensing, and inter-cluster and intra-cluster data communication energy consumptions, and then show that network energy consumptions can be minimized by determining an optimal number of clusters that balances energy consumption for inter-cluster and intra-cluster communications. The problem of nodes to choose their respective optimal clusters is formulated as a Markov decision problem and the results obtained show that the algorithm is capable of adapting to a dynamic environment and converging to an optimal solution.

We also showed that pairwise constraints can be implemented in spectrum-aware clustering to improve primary user detection. Also the energy cost and local decision accuracy have a significant influence on determining the optimal cluster. We further showed through simulations the performance improvement of our approach over groupwise constraint-based algorithms in terms of energy efficiency, channels sensing performance and computational complexity, which are vital to resource constrained devices such as CR-WSN.