Learning Human Strategies for Tuning Cavity Filters with Continuous Reinforcement Learning

Abstract

Learning to master human intentions and behave more humanlike is an ultimate goal for autonomous agents. To achieve that, higher requirements for intelligence are imposed. In this work, we make an effort to study the autonomous learning mechanism to solve complicated human tasks. The tuning task of cavity filters is studied, which is a common task in the communication industry. It is not only time-consuming, but also depends on the knowledge of tuning technicians. We propose an automatic tuning framework for cavity filters based on Deep Deterministic Policy Gradient and design appropriate reward functions to accelerate training. Simulation experiments are carried out to verify the applicability of the algorithm. This method can not only automatically tune the detuned filter from random starting position to meet the design requirements under certain circumstances, but also realize the transfer of learning skills to new situations, to a certain extent.

1. Introduction

As dominators of the earth, human beings have higher intelligence than any other biological species. Humans are skilled in summarizing rules from observed phenomena, forming knowledge, and applying it to understanding new things. Throughout history, the evolution of humans is basically the process by which new methods and techniques are created to alleviate human labor, thus promoting the development of science and technology. As one of the greatest inventions in the 20th century, robotics has made great progress since the late 1950s. The emergence of robots is the inevitable trend of social and economic development, and its rapid growth has been improving the level of social production and the quality of human life.

Although people are eager to have robots in their homes, there is no doubt that the impact of robots on manufacturing is even greater, especially in the last decade when traditional manufacturing has undergone unprecedented changes. Due to the yearning for a better life, many people are unwilling to choose dangerous or boring jobs, which makes different types of human jobs face labor shortages. In this context, industrial robots show extraordinary ability in replacing human labor and improving productivity. However, some tasks that rely heavily on human dexterity or intelligence remain difficult to replace by robots. On one hand, while most industrial robots are adept at repetitive tasks, they are not flexible enough to manipulate forces, especially for some precision or fine machining; on the other hand, most robots today and fully automated production lines are not very good at understanding human intentions or learning sequential strategies from human behavior. Therefore, these automated devices are very sensitive to changes in context. To solve these problems, intelligent learning algorithms and human demonstrations aimed at improving robot autonomy have been studied [1,2].

In this work, we mainly consider the second aspect mentioned above. The tuning process of microwave cavity filter is studied in detail. Tuning is the last step of the cavity filter before it leaves the factory. It is usually a boring and complicated work, which is completed manually by experienced tuning technicians. Our goal is to automate tuning.

The cavity filter is a metal device for filtering signals and suppressing noises, which is often used in microwave, satellite, radar, electronic countermeasure (ECM), and various electronic testing devices. A cavity filter is mainly composed of cavities with resonators, inserted tuning screws, locking screw-nuts, and a covering plate; see Figure 1a. A cavity filter is a product that passes or eliminates microwave signals in specific frequency bands. Due to high working frequencies and inevitable manufacturing errors, unfortunately, almost every cavity filter product needs to be manually adjusted to meet its design specifications. During the tuning process, the measured cavity filter is connected to a Vector Network Analyzer (VNA) (Figure 1b) and iteratively adjusted by a technician with a screwdriver. Almost every inserted screw will be tuned according to the S–parameter curves shown in the VNA. The curves indicate the current tuning state of the product and give guidance to the tuning technician to perform the next action such that the curves are gradually optimized until reaching the desired targets. Figure 2 shows the overall manual tuning process.

There are numerous possible difficulties during the task of cavity filter tuning. First, the relations between tuning screws and S–parameters would be complex. Theoretical analysis does not ensure validity since the manufacturing errors are not considered. In fact, each screw has its physical function in adjusting the S–parameter curves, but the actual product differs a lot from the designed pattern. Second, the cavity filter products have the feature of small-batch multiple varieties, which means that the tuning strategy for one filter category is not necessarily valid for another, even though they may have similar structures or tuning elements. Even for the same product type, differences would be obvious between individuals. Therefore, it is futile to simply copy the inserted screw positions of a tuned filter to a detuned one.

The above challenges make experienced technicians very critical. These “experts” are usually trained for several months to master the tuning strategies before they can handle real tuning tasks. An untrained person may get lost in the mazelike tuning task and never reach the target. On the basis of our investigations, experienced tuning technicians differ in two aspects from ordinary people with no experience. The first significant difference is that experienced technicians can accurately identify and evaluate the current situation, i.e., whether a curve’s change is good or bad. This is the foundation of all success. On this basis, they are also expertized in mastering the tuning amount. In other words, they know which screw to tune, and to what extent. However, all of these experiences are usually perceivable but indescribable.

To automate the tuning process, a powerful and stable automated tuning robot is essential. More importantly, a tuning algorithm should be designed that can accurately provide a tuning strategy at each tuning stage to successfully tune the filter as quickly as possible. In previous studies, automatic tuning methods based on time-domain response [3], model parameter extraction [4], and data-driven modeling techniques (i.e., Neural Network [5], Support Vector Regression [6], fuzzy logic control [7], etc.) have been investigated; however, these methods either rely heavily on individual product models or require data collection, and have limited generalization capabilities. In this work, we still focus on the tuning algorithm but pay more attention to the autonomy of the algorithm. To easily implement the algorithm, a computer-aided tuning robot system has also been developed, but the details are not the focus of this article.

As a breathtaking achievement in recent years, Reinforcement Learning (RL) has attracted extensive attention worldwide. From the perspective of bionics and behavioral psychology, RL can be seen as a manipulation of the conditioned reflex. Thorndike proposed the Law of Effect, which indicates that the strength of a connection is influenced by the consequences of a response [8]. Such a biological intelligence model allows animals to learn from the rewards or punishments they receive for trying different behaviors to choose the behavior that the trainer most expects in that situation. Therefore, the agent using RL is able to autonomously discover and select the actions that generate the greatest returns by experimenting with different actions. Intuitively, the manual tuning process is very similar to RL in many ways. To tune the screws of a cavity filter is equivalent to exploring the environment while watching the S–parameters curve from the VNA, which relates closely to obtaining environmental observations. The reward is instantiated in the personal evaluation of the current situation. For these reasons, plenty of studies have exploited RL for automatic tuning [9]. In this work, we utilize an actor–critic RL method to search for the optimal tuning policy.

Compared with the related work, the contributions of this paper are as follows. First, instead of simply fitting the screw positions to the S–parameters curves, we paid more attention to the process of tuning. In particular, the cavity filter tuning problem was formulated as a continuous RL problem and an effective solution was proposed based on Deep Deterministic Policy Gradient (DDPG). Second, we appropriately designed a reward function inspired by the experienced tuning processes, which stabilizes and accelerates the RL training. The proposed method was tested on a real cavity filter tuning issue and positive results were received. Third, we presented a framework to transfer the learned tuning policy to a new detuned product individual of the same type. With limited steps of exploration on the new product, a data-driven mapping model was built, and the relationship between the new and the old product could be learned. With this framework, the policy learned from RL can be generalized.

The remainder of the article is structured as follows. Section 2 introduces the tuning method of cavity filter and the basic principles of RL. In Section 3, our approach to autonomic tuning using continuous reinforcement learning is developed and presented. We first illustrate the framework of the algorithm and then explain the reward shaping through human inspirations. We also consider the generalization issue to make the algorithm more adaptable. A set of simulation experiments are carried out and the results to verify the validity of the framework are given in Section 4. In Section 5, some relevant issues are discussed in depth. Finally, Section 6 concludes the paper and makes a plan for future work.

2. Related Work

2.1. Automatic Cavity Filter Tuning

Research on automatic tuning algorithms has been going on for a long time. Studies in [3,10] utilized the time-domain response of the input reflection characteristics of the filter. This technique requires an optimal response as a template and relies on accurate estimates of features, which is not realistic in some cases. A sequential adjusting method based on Group Delay was proposed by [11]. This technique depends on the hypothetical computation of the group delay for each resonator. Notwithstanding, when processing the group delay of some resonator, other resonators should be short-circuited, which is infeasible in practice. Other methods based on model parameter extraction [4], poles and zeros [12], coupling matrix decomposition and extraction [13], etc. are also theoretical but almost invalid in most practical cases.

Another set of technologies utilize state-of-the-art artificial intelligence. Michalski proposed a novel tuning strategy building a data-driven model for the cavity filter product [5]. A feed-forward neural network was constructed to model the relationship between S–parameters and the screw deviations $\Delta Z$, i.e., to train a nonlinear mapping $f:S\to \Delta Z$. When tuning a detuned filter, the model is used to predict the screw deviations so that the desired S–parameters are approximated. Similarly, Zhou et al. exploited Support Vector Regression (SVR) to construct metamodels between screw rotations and coupling matrix change [6,14]; the screw rotations are not measured but obtained from two optimization procedures, and the calculation of coupling matrix also brings uncertainties. Furthermore, fuzzy logic control [7,15], neural-fuzzy control [16], Extreme Learning Machine (ELM) [17], and regularized deep belief network (R-DBN) [18] have also been applied in modeling cavity filters. These data-driven methods learn models simply from pregathered data without analyzing the complex theoretical and physical characteristics of the filter product, eliminating the error produced by the difference between the ideal model and the real product. However, these data-driven modeling techniques have constraints in generalization. In other words, to build a data-driven model of a cavity filter, a large number of data pairs should be collected; further, for every filter product type, data pairs should be recollected. Furthermore, the trained model cannot always adapt to the difference between individuals of the same product type, which limits the practicability. Wu et al. proposed new methods by integrating intelligent algorithms with filter theories, which also had good tuning effects [19,20], but the essence of the method is still fitting the screw and the scattering parameters, which might have the same problem as previous methods.

Different from the literature, we do not focus on any specific cavity filter model but make more efforts to study the tuning process. The tuning process of an expert is a typical exploration and exploitation process. The tuning expert interacts with the environment by adjusting the screws and observes the resulting S–parameter curves from the VNA. Then, a new tuning action is taken and the S–parameters are changed accordingly. Iterations are repeated until the filter is fully tuned. In [9], reinforcement learning (RL) approach was first tried to model the tuning process in the framework of Deep Q-Learning (DQN). However, this work only considered two screws, and the action was discretized for Q-learning. In [21], a novel automatic fine-tuning approach using DQN was proposed for planar bandpass filters. A forward model was trained by supervised learning to construct the environment and then interact with the agent in a DQN setting. As an improvement of DQN in handling with continuous state and action spaces, DDPG has been applied for tuning cavity filters with more tuning elements [22]. In [23], a new learning algorithm called Deep Imitation-Reinforcement learning for Tuning (DIRT) was proposed. By using the policy learned from behavior cloning, DIRT improved the performance of DDPG.

Based on the past achievements, in this work, we also explore the extension of RL algorithms in the tuning task with a carefully-designed reward function to accelerate training.

2.2. Reinforcement Learning and Reward Shaping

Reinforcement learning (RL) is a highly intelligent machine learning method. It does not rely on specific supervision, but purely on the learning strategy of starting from scratch by exploring the environment. RL has been proposed and created for some time, and more and more application domains see the key role of RL. However, most RL algorithms are still in their infancy and show very limited possibilities for real-world practical tasks. The high dimension of action space and state space and the difficulty of convergence hinder the development of RL. Represented by Deep Q-Learning (DQN), the combination of deep learning and RL has changed these. DQN uses a deep structure for value function approximation and designs two important techniques called the Experience Replay and the Target Network to make the learning more stable [24,25]. Since then, the study in [26] proposed DDPG algorithm to handle continuous action spaces. In DDPG, the Experience Replay and the Target Network are preserved from DQN, but DDPG uses the actor–critic mechanism for simultaneously updating a policy network and a value network.

RL is always time-consuming. In most practical situations, finding the optimal strategy through a lot of trial and error is unacceptable. In addition, it is important to evaluate the agent’s behavior during exploration. Since the RL algorithm only updates the strategy according to the reward, the definition of the reward function is of great concern. Reward shaping provides a solution by shaping an additional reward that is often designed by human experts or extracted from human demonstrations. Ng proposed an efficient potential-based reward shaping function and proved that it would not affect the convergence to the optimal policy [27,28]. Long-term studies have been conducted in shaping with human advice or demonstrations [29,30,31].

In this work, we designed a shaping reward inspired by experienced human tuning processes. In particular, the shaping term evaluates the tuning performance according to the changing trend of the S–parameters curve. Therefore, an instant reward will be obtained at each training step, and the reward will not be sparse anymore; thus, the learning is accelerated.

As a whole, the method proposed in this work is also based on RL, which means we consider the tuning process rather than the filter model. Furthermore, we consider effective ways to improve the performance of RL in accelerating and transferring.

3. Intelligent Tuning Based on DDPG

3.1. Preliminaries

The observed states from the VNA are S–parameters curves, as shown in Figure 3. Since the cavity filter is a passive device, all signals passing through it come from external sources. During the test, the cavity filter device under test (DUT) is connected to a VNA through input/output ports. The VNA then sends the signal to the DUT to observe the received signal in the form of scattering parameter, or S–parameter for short. To meet the designed requirements of a cavity filter product, it is necessary to examine some test data, including the Return Loss, the Insertion Loss, the Pass-band Fluctuation, and the Out-of-band Rejection. $S_{ij}$ is the common notation of S–parameters, indicating the energy measured at port i injecting to port j. The return loss of port 1 is represented by $S_{11}$, indicating the energy measured at port 1 after injecting from port 1 and returning to port 1. Similarly, the insertion loss of port 1 is always represented by $S_{12}$, representing the energy measured at port 1 after injecting from port 1 to port 2 and returning to port 1. It is always required that the value of the return loss should be minimized within the passband (always less than or equal to $-20$ dB). In other words, the values of $S_{11}$ in Figure 3 must be under the black dashed benchmark line within the specified frequency range. For insertion losses, the value should be maximized within the same passband (always close to 0 dB).

In this work, we only consider the value of return loss, namely, the $S_{11}$ curve in the passband, as the state of RL, and ignore the other specifications, such as insertion loss. Therefore, the goal of tuning is to adjust the screw to control the $S_{11}$ curve to the target line. It is important to note that our tuning method can be applied to any filter class of radio frequency (RF) devices, including cavity filters, combiners, duplexers, and multiplexers.

3.2. Formulations

The RL method in this work follows the basic Markov Decision Process (MDP) settings. At each time step t, the agent observes a state $s_t$ from the environment. Then, it takes an action $a_t$ to interact with the environment and the state of the agent is transferred to $s_{t+1}$. At the same time, the reward $r_t$ reflecting the current state and action pairs is obtained from the environment or specified by a human teacher. With the transition set $(s_t,a_t,r_t,s_{t+1})$, the agent learns the policy with some optimization process, so that the expected future reward is maximized.

In our previous work, the framework of DQN was applied. A value function Q is defined based on the Bellman equation to compute the expected future reward. Then, the optimal action to take at each time step is the one that maximizes the Q values,

$$Q\left(s_t,a_t\right)=r_t+\gamma \underset{a_{t+1}}{max}Q\left(s_{t+1},a_{t+1}\right),$$

(1)

where $s_t$, $a_t$, and $r_t$ are the state, action, and reward, respectively, at step t; $\gamma$ is a discounting factor. In DQN, a replay memory D is used to save transition sequences state, action, reward, new state. At each time step, a minibatch of transitions will be randomly selected from the replay buffer D for training the Q–network by minimizing the distance between the target Q-network and the current hypothesis model:

$$\underset{\theta }{min}\left(\left.y_i-Q\left(s_i,a_i\right)\right|\theta \right),$$

(2)

where the target is

$$y_i=r_i+\gamma \underset{a^{\prime }}{max}\widehat{Q}\left(s_{i+1},a^{\prime };\theta \right).$$

(3)

Instead of using Deep Convolutional Neural Networks to approximate the value function, we extracted a feature set from the original S–parameters as the state. We follow the Experience Replay and the Target Network of DQN to ensure the stability of the algorithm. We also used the Euclidean distance between the S–parameters curves as a reward function. In particular, if the current curve is getting closer to the target, the reward is large, and vice versa. This definition of this reward is somewhat coarse and should be modified in the light of human experience.

In this work, the tuning algorithm is based on another important RL framework—DDPG—in which continuous actions are valid. Experience replay and Target Q-Network are preserved from DQN. DDPG is based on the actor–critic mechanism, where a critic function $Q(s,a)$ and an actor function $\mu \left(\left.s\right|{\theta }^{\mu }\right)$ are parameterized by two separate neural networks. The actor network is also called the policy network, which updates the policy with policy gradient. The critic network is similar to the value function in Q-learning, which evaluates the action from the actor network by updating the Q-values [26].

We define the tuning task as an MDP with the following settings. At each time step t, the agent is in a state $s_t\in \mathcal{S}$. In the tuning problem, the state $s_t$ is defined as the S–parameters ($S_{11}$) curve (or some extracted features) drawn from the VNA. The agent performs an action $a_t\in \mathcal{A}$ at each time step t to transfer state $s_t$ to a new state $s_{t+1}$. The action is defined as the tuning amount we can take, i.e., the adjustment of each screw element. Unlike our previous work [9], screw lengths are continuous values instead of discrete values, which is more accurate and reasonable. Thus, the action at any time step can be represented as a vector of dimension m, where m is the number of valid tuning screws. The reward function consists of three parts, which take into account the process of tuning target and human experience.

The main algorithm is concluded in Algorithm 1. For details of the DDPG algorithm, readers can refer to [26].

3.3. Reward Shaping Inspired by Human Knowledge

After taking the action $a_t$, a reward $r_t$ will be obtained. The value of the reward is an evaluation of the action taken, and the goal of RL is to learn a policy $\pi \left(s\right)\to a$ so that the expected future reward is maximized. In some cases, the reward value can be derived directly from interaction with the environment, such as playing Atari games. In other cases, the reward function should be defined manually, such as the self-driving problem. Other tasks have reward functions that are difficult to define, such as robot manipulations. In these cases, the reward is always learned by demonstration. In this work, we use the experience of human tuning to manually design the reward function to make RL have better performance.

The reward $r_t$ of each step consists of three parts. $r_t^{\mathrm{goal}}$ directly describes the cost of each step taken by an action. Any step other than reaching the goal is rewarded with $-0.01$. If the action results in an eligible S–parameters curve, i.e., reaching the tuning target, the reward will be added by $+10$.

$$r_t^{\mathrm{goal}}\left(s_t,a_t,s_{t+1}\right)=\left\{\begin{array}{cc}{r}^{\mathrm{target}}\hfill & \mathrm{if}{s}_{t+1}=s_{\mathrm{goal}}\hfill \\ r^{\mathrm{penalty}}\hfill & \mathrm{otherwise}\hfill \end{array}.\right.$$

(4)

Algorithm 1 Intelligent Tuning Based on DDPG
1:	Randomly initialize critic network and actor network with weights $Q\left(\left.s,a\right|{\theta }^Q\right)$ and $\mu \left(\left.s\right|{\theta }^{\mu }\right)$.
2:	Initialize the target network $Q^{\prime }$ and ${\mu }^{\prime }$ with the same weights as above.
3:	Initialize the replay buffer D.
4:	Randomly initialize inserted screw positions.
5:	for episode = 1,M do
6:	Read S–parameters from the VNA and formulate the initial state $s_1$.
7:	for step = 1,T do
8:	Select action $a_t$ according to the current policy and exploration noise.
9:	Tune corresponding screws to execute action $a_t$ and compute reward $r_t$, and observe new S–parameters to formulate new          state $s_{t+1}$.
10:	Store the transition $(s_t,a_t,r_t,s_{t+1})$ in D.
11:	Sample a random minibatch of N transitions $(s_i,a_i,r_i,s_{i+1})$ from D.
12:	Set $y_i=r_i+\gamma Q^{\prime }\left(\left.s_{i+1},{\mu }^{\prime }\left(\left.s_{i+1}\right|{\theta }^{{\mu }^{\prime }}\right)\right|{\theta }^{Q^{\prime }}\right)$.
13:	Update the critic network by minimizing the loss: $L=\frac{1}{N}{\displaystyle {\displaystyle \sum }_{i=1}^N}{\left(y_i-Q\left(\left.s_i,a_i\right|{\theta }^Q\right)\right)}^2$.
14:	Update the actor network using the sampled policy gradient: ${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{N}{\displaystyle {\displaystyle \sum }_{i=1}^N}{\left.{\nabla }_{a}Q\left(\left.s,a\right|{\theta }^Q\right)\right|}_{s=s_i,a=\mu \left(s_i\right)}{\left.{\nabla }_{{\theta }^{\mu }}\mu \left(\left.s\right|{\theta }^{\mu }\right)\right|}_{s_i}$.
15:	Update the target networks:
	$\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}\hfill \\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}\end{array}$
16:	end for
17:	end for

To impose a penalty for one or more screws exceeding the length limit, a total reward of $-0.1$ is added. More specifically, if the action causes the screw to exceed its limit, the screw will remain unchanged and the agent will be rewarded with $-0.1/m$, where m is the number of valid screws. The reward $r_t^{\mathrm{limit}}$ is formulated as

$$\begin{array}{c}{r}_t^{\mathrm{limit}}\left(s_t,a_t,s_{t+1}\right)=\hfill \\ -\frac{0.1}{m}·{\displaystyle {\displaystyle \sum }_{i=1}^m}\mathbf{1}\left[z_{t+1}^{\left(i\right)}\in \left(z_{max}^{\left(i\right)},+\infty \right)\cup \left(-\infty ,z_{min}^{\left(i\right)}\right)\right],\hfill \end{array}$$

(5)

where $\mathbf{1}[·]$ is the indicator function with

$$\mathbf{1}\left[x\right]=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x\mathrm{is}\mathrm{True}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array},\right.$$

(6)

and $z_t^{\left(i\right)}$ denotes the insertion position of the i-th tuning screw at the time step t. $z_{max}^{\left(i\right)}$ and $z_{min}^{\left(i\right)}$ denote, respectively, the maximum and minimum insertion position of the i-th screw.

The third part is the most important reward $r_t^{\mathrm{direction}}$, which is formed in the evaluation of transitions from the current state to the next state, inspired by human tuning experience. Take into account the experienced tuning process. Technicians are able to optimize the S–parameters curve because they can evaluate changes to the curve. In other words, they do their best to make the S–parameters curve always move in the right direction. Therefore, the change direction of the S–parameters curve to the new curve can be expressed as a set of vectors. Each $S_{11}$ curve is truncated first, leaving only the central core area. The two curves are then discretized into the same number of sampling points. Thus, the vectors connecting two corresponding points of the curves can form a vector field. See Figure 4 for more details. In Figure 4a, the vector ${\mathbf{v}}_j^{\left(\mathrm{current}\right)}$ represents the moving direction of the point $A_j$ on the blue curve (current state) to the point $B_j$ on the green dashed curve (a new state).

$${\mathbf{v}}_j^{\left(\mathrm{current}\right)}=\left[\begin{array}{cc}{x}_{bj}-x_{aj}& y_{bj}-y_{aj}\end{array}\right].$$

(7)

Similarly, to represent the changing direction of a curve to a target line, a set of vectors connecting each two corresponding points on the curve and the target line can be applied, as shown in Figure 4, denoted by ${\mathbf{v}}_j^{\left(\mathrm{target}\right)}$.

Therefore, we can use the sum of each cosine value of the angle ${\alpha }_j$ between ${\mathbf{v}}_j^{\left(\mathrm{current}\right)}$ and ${\mathbf{v}}_j^{\left(\mathrm{target}\right)}$ to calculate the directional difference of two S–parameters curves, i.e.,

$$cos{\alpha }_j=\frac{{\mathbf{v}}_j^{\left(\mathrm{current}\right)}·{\mathbf{v}}_j^{\left(\mathrm{target}\right)}}{\left|{\mathbf{v}}_j^{\left(\mathrm{current}\right)}\right|\left|{\mathbf{v}}_j^{\left(\mathrm{target}\right)}\right|},$$

(8)

where

$${\alpha }_j=〈{\mathbf{v}}_j^{\left(\mathrm{current}\right)},{\mathbf{v}}_j^{\left(\mathrm{target}\right)}〉.$$

(9)

The intuition to use cosine values is pretty simple. In order to adjust screws to maximally drive the current S–parameters curve to reach the target, we want each point on the curve to move as far as possible toward the goal. The cosine value of two vectors within the range $[-1,1]$ describes the similarity between the current direction and the target direction. The closer the current vector is to the target, the higher the value. More specifically, the cosine value of angle ${\alpha }_j$ in Figure 4a is larger than 0 because ${\alpha }_j$ is less than 90${}^{\circ }$, representing that ${\mathbf{v}}_j^{\left(\mathrm{current}\right)}$ and ${\mathbf{v}}_j^{\left(\mathrm{target}\right)}$ almost share the same direction. Conversely, the value of angle ${\alpha }_j$ in Figure 4b will be less than 0 since ${\alpha }_j$ is larger than 90${}^{\circ }$, which indicates that the two vectors have opposite directions. Therefore, the third shaping reward is formulated as

$$r_t^{\mathrm{direction}}\left(s_t,a_t,s_{t+1}\right)=\sum _{j=1}^{n}cos{\alpha }_{t,j},$$

(10)

where

$$cos{\alpha }_{t,j}=\frac{{\mathbf{v}}_j^{\left(t\right)}·{\mathbf{v}}_j^{\left(\mathrm{target}\right)}}{\left|{\mathbf{v}}_j^{\left(t\right)}\right|\left|{\mathbf{v}}_j^{\left(\mathrm{target}\right)}\right|},$$

(11)

in which ${\mathbf{v}}_j^{\left(t\right)}$ denotes the vector connecting the point j on the $S_{11}$ curve before and after tuning at time step t.

The cosine value only describes the direction of movement of each point on the curve. In addition, the length of each vector represents the extent of the movement. So, we multiply the cosine value by the moving amount of that point in the target direction. Therefore, the human-experience-shaped reward can be improved by

$$r_t^{\mathrm{direction}}\left(s_t,a_t,s_{t+1}\right)=\frac{1}{n}\sum _{j=1}^n\frac{\left|{\mathbf{v}}_j^{\left(t\right)}\right|}{\left|{\mathbf{v}}_j^{\left(\mathrm{target}\right)}\right|}·cos{\alpha }_{t,j}.$$

(12)

Therefore, the generated shaping reward inspired by human experience naturally shows the overall degree to which the curve changes in the direction of the target. The overall reward function is the sum of these three parts:

$$r_{t+1}={\lambda }_1·r_{t+1}^{\mathrm{goal}}+{\lambda }_2·r_{t+1}^{\mathrm{limit}}+{\lambda }_3·r_{t+1}^{\mathrm{direction}},$$

(13)

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are weights for a trade-off.

3.4. Generalize the Learned Policy

To learn how to tune a filter with RL is only a first and basic step. Since RL training always takes a long time to converge, and every product individual is different, how to adapt the individual differences is a challenging issue. If the trained policy cannot be generalized, RL loses its significance and will have no more advantage than supervised learning. In this work, we present a framework to tackle the generalization problem after RL training. Specifically, our aim is to tune a new detuned cavity filter to its target within as few steps as possible, after RL training on a benchmark product of the same type.

The framework contains two phases: learning and transferring. In the learning phase, we randomly explore the new filter product, collect enough data pairs, and build a data-driven mapping model between the new filter product and the old one. To achieve this, an inverse estimation model needs to be first built, i.e., a mapping from the S–parameters to the screw positions of the old filter product. The inverse estimation model could be built with data examples collected during the RL training process. Once we have a new product individual, we first randomly place the screws and record their corresponding S–parameters curves. Then, the inverse estimation model is used to predict the screw positions on the old filter product (which has already been trained). Finally, the two sets of screw data are used for constructing the data-driven mapping model. The learning phase is concluded in Figure 5a.

In the transferring phase, the learned data-driven mapping model is used to transfer the trained RL policy to the new product. When connecting the new product to the VNA, S–parameters could be measured. The trained RL policy network $\mu \left(s\right)$ predicts the best action to take. However, since the policy network is based on the old filter product, the predicted action cannot be directly applied. The mapping model learned in the learning phase should be used to transfer screw positions from the old to the new. The action on the new product is executed. The process is shown in Figure 5b.

It is worth noting that to build the inverse estimation model, data pairs including screw positions and S–parameters curves should be collected. Fortunately, the data could be collected at the same time with RL training. A way to correctly record screw positions is necessary.

4. Experiment

4.1. Settings

To evaluate the proposed tuning algorithm, simulation experiments are carried out on a four-screw cavity filter. Since frequent adjustment of screws on real products can cause damage to filter products and the tuning robot, we use a data-driven simulation model for experiments instead. The screw positions and S–parameters data collected randomly by the automatic tuning robot were used to build the model (Figure 6). In total, 2400 data pairs were collected. The robot tuning system is composed of a three-axis robotic arm, an industrial personal computer, and a control cabinet. Five servomotors are used for moving in the x, y, and z-direction, as well as the rotation for tuning the screw and the screw-nut. The robot can output specified rotation angles, so that desired tuning amount on the filter can be recorded. The $S_{11}$ curve is observed from the VNA as the state for RL. To calculate the shaping reward, the central region of the $S_{11}$ curve is first truncated. The cavity filter model is a combiner with four channels, and only one channel is considered for the test (Figure 7). The channel has four cavities with seven tuning screws, but for the sake of simplicity, only the tuning screws of four resonators are used in the experiment, indicating that the action dimension of RL is 4. Other screws remain in their standard position.

The DDPG model used in our experiments is similar to the low-dimensional network structure in [26]. We extracted the S–parameters into a 20-dimensional feature set with Principle Component Analysis (PCA) as the input to the value network (critic) and the policy network (actor). The output of the policy network was a 4-dimensional set of continuous actions, representing actions taken on four screws. Both networks are feed-forward neural networks with the extracted feature set as the input. Both of them have two hidden layers, with 300 units on the first hidden layer and 600 units on the second. The activation function for the two hidden layers is Relu, while the output layer uses tanh as the activation function. The learning rate is 0.0001 for the actor network and 0.001 for the critic network. Weights are randomly initialized. For the filter product we used, the effective screw insertion position values were normalized and limited to $[-3,+3]$, representing the rotation performed to the screw from a standard position by $[-540,+540]$ degrees. For each tuning step, the action value output by the policy network is multiplied by a small size length to calculate the final tuning amount on each screw, i.e., the update of the screw positions would be

$$z_{t+1}^{\left(i\right)}:=z_t^{\left(i\right)}+a_t^{\left(i\right)}·\Delta z^{\left(i\right)},$$

(14)

where $z_t^{\left(i\right)}$ is the position of the i-th tuning screw at time step t, $a_t^{\left(i\right)}$ is the output action of the policy network at time step t, and $\Delta z^{\left(i\right)}$ is a small step-size. The $\Delta z^{\left(i\right)}$ of all four screws were set to the value of 0.25 in the experiment of this proposed work.

To avoid getting stuck, we only trained the DDPG model with a maximum of 200 steps per episode and tested the model after every four episodes. In other words, if the algorithm has already tuned the filter 200 times but the filter has not yet been fully tuned, this episode will be stopped. At the beginning of each episode, the screw insertion positions were randomly initialized. The ratios for the three reward terms were set with ${\lambda }_1={\lambda }_2=1$, ${\lambda }_3=10$. The ratio values were selected based on experience or practical tests. $r^{\mathrm{target}}$ and $r^{\mathrm{penalty}}$ were set to $+10$ and $-0.01$, respectively. Other parameter values related to RL training are listed in Appendix A.

To validate the effectiveness of the proposed, human-inspired reward shaping, three sets of experiments with different reward settings were conducted. The three cases are (1) with the human-inspired shaping reward $r_t^{\mathrm{direction}}$, (2) with a Euclidean distance shaping reward $r_t^{\mathrm{distance}}$, and (3) without any shaping reward. Specifically, $r_t^{\mathrm{distance}}$ measures the distance between the S–parameter values and the tuning target:

$$r_t^{\mathrm{distance}}\left(s_t,a_t,s_{t+1}\right)=\left\{\begin{array}{cc}1& \mathrm{if}{\delta }_t>{\delta }_{t+1}\\ 0& \mathrm{otherwise}\end{array},\right.$$

(15)

where ${\delta }_t$ is the Euclidean distance between the $S_{11}$ values and the return loss within the passband:

$${\delta }_t=\sqrt{\sum _{k=1}^{n_{pb}}{\left(S_{11}\left(t,k\right)-RL\right)}^2}\mathrm{with}{S}_{11}\left(t,k\right)-RL>0,$$

(16)

where $RL$ is the return loss target (e.g., $-20$ dB), $n_{pb}$ denotes the number of points sampled on the curve within the passband (only considering the points where $S_{11}>RL$). The reason to use $r_t^{\mathrm{distance}}$ for comparison is that the Euclidean distance is intuitive and simple for evaluating the curve state, and has been used in previous related work [9,22]; however, it has limitations in handling the horizontal shift of the curve, which will be analyzed later.

The algorithm was implemented in Python language and compiled in Anaconda. The environment of RL was a neural network model, pretrained by Keras in Anaconda with randomly collected data pairs. Some parameter values related to RL training are listed in Appendix A.

4.2. Results

Figure 8 describes the corresponding results in three cases, respectively. The figures on the left show the tuning steps during training, while the figures on the right show the tuning steps during testing (i.e., simply run after every four training episodes). The vertical axes show the number of tuning steps in each episode. At the beginning of the explorations, the agent can hardly make the filter be fully tuned in 200 steps. As the exploration goes on, the agent takes fewer steps to succeed. In other words, the fewer steps in one episode, the better; further, the fewer episodes used to converge, the better. When applying the knowledge-inspired shaping reward $r_t^{\mathrm{direction}}$, the algorithm almost succeeded in finding the optimal tuning strategy in about 100 episodes. In contrast, using the Euclidean distance reward $r_t^{\mathrm{distance}}$ and without any shaping reward (i.e., r consists only of $r^{\mathrm{goal}}$ and $r^{\mathrm{limit}}$), it takes more than 250 episodes for the algorithm to converge to an optimal policy.

The results indicate the following: (i) By applying the DDPG algorithm with and without reward shaping, the cavity filter can both be fully tuned. This indicates that the proposed RL method is effective to learn how to tune a detuned filter to its target from scratch. (ii) By using a carefully prescribed shaping reward $r_t^{\mathrm{distance}}$, the RL agent takes fewer episodes to converge to the optimal policy than using no shaping reward. This is because, without any shaping reward, the agent can only receive feedback when the cavity filter is finely tuned, which always takes a long period of time. On the contrary, adding an extra shaping reward provides instant feedback at each training step; thus, the total reward will not be sparse anymore and the learning is accelerated. (iii) An inappropriate shaping reward will not accelerate learning. Simply computing the Euclidean distance to the target for shaping the reward has no obvious advantage over without any shaping reward.

Although it seems reasonable to use the distance for evaluating the tuning state, it is not good at handling the cases when the S–parameters curve has a horizontal shift. Figure 9 illustrates the computation of the Euclidean distance of three different S–parameter curves. Only the points above −20 dB are considered for computing the distance, which have been marked red. In Figure 9a, the curve is shifted away from the target vertically. A total of 114 points are included for calculation, and the computed Euclidean distance is 76.11. In Figure 9b, the curve almost reaches the target, and only 13 points are included for calculation, with a final distance value of 4.21. In Figure 9c, the curve also seems to be approaching the target and only 18 points do not meet the requirement, but the corresponding distance value reaches 51.11, which is relatively high. Compared with the curve in Figure 9b, the shape of the curve in Figure 9c is almost the same, but has a slight shift to the right. This indicates that a small horizontal shift will cause a large distance value. Therefore, simply using the Euclidean distance for evaluating the reward is not always appropriate.

Figure 10 shows the learning curves of the RL process. Figure 10a shows the accumulated rewards. We can find that with knowledge-inspired reward $r_t^{\mathrm{direction}}$ or the Euclidean distance reward $r_t^{\mathrm{distance}}$, the reward value is accumulated faster than without any shaping reward because the agent obtains shaping reward values more frequently, and a sparser reward value of $+10$ will be added if the filter is successfully tuned. A steep rise in accumulated reward denotes a more frequently added $+10$. We also reserved a set of randomly obtained states and computed their average Q values after each training episode. The blue and green curve in Figure 10b increase more quickly and steadily than the red one, which also reflects the effectiveness of the shaping reward. However, when using $r_t^{\mathrm{distance}}$, the total number of training steps reaches about 55,000, which indicates more failed episodes (i.e., 200 training steps have been fully used but the filter has not been fully tuned) during training. Therefore, using an inappropriate shaping reward does not accelerate training but reduces the exploration performance.

Some S–parameters curves are also obtained during the experiment. In Figure 11, the curves of $S_{11}$ for four different tuning processes are given. Most tuning tests could be finished in about 10 steps (depending on the starting position and the step size). As the target of RL is not a fixed $S_{11}$ but a range, the end states of the experiment are different in shape. In other words, the shape of the final curve does not matter. Any curve is acceptable as long as the value of $S_{11}$ in the passband is less than $-20$ dB.

4.3. Generalization Test

Finally, the generalization framework proposed in Section 3.4 was tested. We manually constructed new filter products based on the trained product by making alterations to the screw positions. Specifically, we linearly scaled each screw of the old cavity filter by multiplying with a set of factors $[0.5,1.0,1.5,0.25]$ to simulate a new one. Then, the mapping model was learned as the learning phase described in Section 3.4. It is important to reduce the number of collected data pairs because frequently adjusting the screws on a new filter product should be avoided. We collected 50, 100, and 1000 data examples, respectively, on the new filter product for training and comparison. Afterwards, the trained mapping models Mapping 50, Mapping 100, and Mapping 1000 were applied directly in the transferring phase. At each tuning step, the action is predicted by the RL policy network and then mapped to the corresponding value in the new filter. Then, the mapped action is executed on the new filter product.

For each of the three models—i.e., Mapping 50, Mapping 100, and Mapping 1000—100 episodes with a maximum of 50 tuning steps in each episode have been tested. Figure 12 provides some test results. In Figure 12 (left), the number of tuning steps in each episode for each mapping model has been shown. Figure 12 (right) quantitatively compares the three mapping models. With Mapping 50 model, only in 65 episodes among 100 could the new constructed filter be successfully tuned, while with Mapping 1000 model, 82 episodes are target-reached. It is evident that with more training data, the new product could be more accurately transferred to the old trained model. In practice, to find a good trade-off between data size and performance should be carefully treated.

It should also be noted that, in this work, we simulate the new filter product by simply rescaling the screws of the old. However, constructing other kinds of differences should also be possible, such as by shifting or adding noises to screws of the old one. In practice, the differences between individuals could be more nonlinear and complicated, which is also another important topic to be studied in the future.

5. Discussions

It seems a fuss to use RL, since other methods, such as training supervised learning models [5,6,14,16,32] or to study the physical characteristics of the cavity filter [3,10,11,12], can still achieve the same goal; in most cases, those methods also do not need a large number of iterations. However, these methods always need a pretuned filter model as a benchmark or experienced tuning experts for guidance. In addition, they have weak potential in generalization. As explained in Section 1, cavity filters have a wide range of types, and they are different in structure and design specifications. Therefore, simply modeling one filter product may fail in adapting to others. In contrast, humans can learn strategies from experience and extract useful information for further use, no matter how the filter type changes. Therefore, it is more reasonable to study the tuning strategies rather than a cavity filter sample.

RL algorithms do not depend on any explicit guidelines or supervision but autonomously learn the tuning strategies from scratch. More importantly, the learned strategies are expected to be generalized to other cases, i.e., to different individuals of the same type or even totally different types. This work is an extension to [9,33]. In [9], DQN is used for the first time to solve the tuning problem but the state and action spaces are both very limited. In [33], DDPG first shows its effectiveness for tuning. The methods proposed in [21,22,34] also utilize DQN, double DQN, or DDPG, and based on previous studies, these works attempt more filter orders (screw numbers) or more elaborate reward functions, but have little consideration of generalization or transfer problems. In this work, we have considered the issue of transferring the learned knowledge, even though the differences between individuals are man-made, which does not necessarily accord with reality. Therefore, the problem related to generalization or transfer is still demanding work to be addressed in the future. It is also worth noting that some other methods such as Particle Filtering and Particle Swarm Optimization [35] can also be considered in our future work to focus on the tuning process.

Our current study is somewhat limited by the difficulty of the experiment. Although we have a robotic tuning system, it is not enough efficient, and thus, has not been directly used for executing RL training for safety reasons. Without an effective tuning machine, either a simulation model or manual tuning can be used for the experiment. By comprehensive consideration, we use the robot tuning system for collecting data pairs to build the simulation environment, then train the RL model in simulation.

Another popular way to integrate human experience with RL is to use imitation learning (also called learning from demonstration), which provides a straightforward way for the agent to master flexible policies. Studies in [36,37] integrate DQN and DDPG with human demonstrations by pretraining on the demonstrated data. However, this idea is somewhat hard for our task. First, in order to use demonstration data, a tuning expert should be always available, which is not guaranteed. Second, data acquisition is also important. The tuning action is continuous, and the tuning amount directly determines the change amount of the S–parameters. To record the demonstration data, both screw positions and S–parameters should be discretized; it also needs collaboration with tuning technicians. Finally, the recorded data might highly depend on the skills of the tuning technician. Different people perform differently, and even the same person may differ in the process of tuning the same product, which brings additional noise. Due to these reasons, imitation learning is currently not appropriate for this task. Instead of directly using human demonstration, we integrate human intelligence into the design of the reward function, which performs much better than simply learning from human demonstrated data.

6. Conclusions

Humans can handle plenty of complicated tasks that are difficult for robots. How to extract and learn from human strategies is a key issue for making robots more intelligent. Tuning cavity filters generally relies on the rules of thumb and can only be operated by highly skilled technicians. The automation of cavity filter tuning is a challenge. In this paper, the tuning task of cavity filters is defined as a continuous RL problem, and a possible solution for autonomous tuning is presented under the DDPG framework. We designed an appropriate reward function with a shaping term inspired by the human tuning experience. At any time, the S–parameters curve is driven in the desired direction as much as possible. We successfully verified our algorithm on a four-screw cavity filter product. The algorithm can adjust the filter to the desired target from any random starting position of the screw within a certain range of screw positions. We also proposed a framework for generalizing the trained policy on a new detuned product individual and tested its validity on products with simulated differences.

Nevertheless, the proposed method for automatically tuning cavity filters still needs to be extended. The filter model used in the experiment is simple and the curve always changes in the vertical direction. The environment used for DDPG training was a simulation model rather than a real product, which also leaves room for improvement. The simulation environment cannot be completely consistent with the real situation and cannot accurately reflect individual differences when tackling the generalization problem. One possible future direction is to try more tuning screws and a wider range of screw positions, with the help of a powerful tuning robot. Another future orientation is to continue to explore the transfer problem, which is to consider the transfer of learned skills to more complex but more practical new environments. It is also considerable to use domain randomization, i.e., to train the RL model directly on different filter models so that the generalization ability of the trained policy could be increased.