An improved whale optimization algorithm for UAV swarm trajectory planning

Abstract

For the problem of trajectory planning in the small-scale unmanned aerial vehicle (UAV) swarms, classical intelligent algorithms and newly emerged bio-inspired algorithms often suffer from being trapped in local optima, leading to suboptimal solutions. Moreover, these algorithms fail to ensure optimal solutions and convergence speed in high-speed dynamic UAV networking scenarios. In response to these challenges, this paper proposes an Improved Whale Optimization Algorithm (IWOA) that considers the global perspective. To address the issue of initial solution diversity, an opposition-based learning method is introduced by constructing a reverse population, enhancing the diversity of the initial population. To overcome the problem of the algorithm getting trapped in local optima, random convergence factors, and end-point random neighborhood perturbations are incorporated to accelerate the global convergence speed of the IWOA and prevent it from getting stuck in local optima. Additionally, a grid digital elevation model is employed when constructing the experimental environment model, considering various physical constraints of the UAV swarm. This ensures that the simulation validation of the IWOA algorithm is closer to real-world scenarios. Simulation results indicate that, compared to commonly used algorithms, the IWOA can generate more optimal trajectories for drone swarm planning under constraints resembling real-world scenarios. It exhibits better performance in trajectory evaluation, convergence speed, and stability, thereby meeting the requirements of trajectory planning for small-scale UAV swarms. The proposed IWOA enhances UAV swarm coordination and efficiency, significantly impacting real-world applications in various fields.

1 Introduction

UAV technology has attained a high level of maturity and is now extensively utilized across diverse fields. In the military field, as an indispensable part of the information-based combat system, it is foreseeable that UAVs will carry out more types of intensive tasks in the future battlefield and become a “multi-faceted player” in the battlefield. The trajectory planning of UAVs is an important issue in UAV applications. Currently, common trajectory planning algorithms for various scenarios and specific challenges include A* algorithm [1–3], D* algorithm, PRM algorithm [3], artificial potential field method [4], RTT algorithm [5], Dijkstra’s method [6], and others. However, traditional path planning methods face challenges such as high dimensionality, obstacle handling, dynamic environments, constraints, and others when addressing 3D path planning issues, necessitating the development of more efficient and adaptive algorithms to overcome these challenges [7]. Many traditional deterministic planning algorithms suffer from problems such as local convergence and high computational complexity in complex environments, which limits their application in UAV trajectory planning. The emerging class of intelligent optimization algorithms has gradually become a practical solution with the development of random search theory, including methods such as particle swarm optimization [8], genetic algorithm [9], ant colony optimization [10], differential evolution [11], firefly algorithm [12, 13], grey wolf algorithm [14], bat algorithm [15], and others. However, due to the complexity of problems and the diversity of search spaces, these algorithms risk converging to local optima. The Whale Optimization Algorithm (WOA) [16] is a type of swarm intelligence algorithm that solves optimization problems by simulating whale behavior. Its simple implementation and efficient search performance make the Whale optimization algorithm a new research focus in the field of optimization algorithms. A Levy flight strategy and information exchange mechanism were introduced during the iterative process of the Whale Optimization Algorithm [17], enabling faster convergence and optimal trajectory cost considerations under threat and constraint conditions. Still, they may lead to ignoring the locally optimal solution when the step size is too large. Literature [18] proposes a multiple population improved whale optimization algorithm (MI-WOA), which divides the population into better and worse groups. In order to improve the algorithm’s development ability and solution accuracy, better individuals are used for development. Worse individuals are used for exploration, which ensures the diversity of the population and improves the algorithm’s exploration ability. However, in the late stage of the algorithm, there is the problem of slow optimization search. Literature [19] proposes a hybrid algorithm (WOA-AEFS), which combines WOA, artificial bee colony algorithm, and firefly algorithm, and aims to avoid premature convergence by introducing the exploration influence rate and adjusting the position updating method to realize the balance between the exploitation and exploration ability. However, the complexity of this algorithm is high, and solving large-scale global optimization problems has yet to be verified. Reference [20] introduces the MWOA algorithm, which combines tent map chaos mapping, adaptive chaos search strategy, and tournament selection strategy [21–23]. It reduces the likelihood of falling into local optima, enhancing the algorithm’s robustness. Nevertheless, higher time complexity is challenging when addressing low-dimensional problems.

The Improved Whale Optimization Algorithm (IWOA) proposed in this work is specifically designed to address the trajectory planning challenges for small UAV swarms, particularly those related to local optimization. In traditional UAV trajectory planning algorithms, local optimization problems frequently arise due to the complexity of high-dimensional search spaces, the deterministic nature of these algorithms, and the often insufficient diversity in the solution population. These factors can cause algorithms to become trapped in local optima, where they cannot explore more globally optimal solutions. As a result, the generated trajectories may be suboptimal, leading to increased mission time, higher energy consumption, and potentially greater risks of mission failure [24, 25].

To mitigate these issues, the IWOA incorporates several strategies to enhance global exploration while minimizing the chances of premature convergence to local optima. Firstly, the inverse learning method is applied to address the initial solution discrepancy by generating a more diverse and optimal initial population when combined with the elite population. This step is crucial, as a poor initial solution can significantly hinder the algorithm’s ability to escape local optima, particularly in complex environments. Furthermore, to counteract the local optimal trap commonly encountered in conventional algorithms, the IWOA introduces random selection of the convergence parameter factor and the addition of random perturbations near UAV nodes. This increases the “activity” or variability in the exploration process, allowing the algorithm to search a broader area of the solution space, thereby reducing the risk of becoming stuck in local optima. By ensuring that the number of iterations does not constrain the convergence factor, the IWOA maintains a dynamic balance between exploration and exploitation throughout the optimization process. As a result, the IWOA algorithm accelerates global convergence while effectively preventing premature convergence to local optimal solutions. The simulation and test results demonstrate that IWOA not only achieves faster convergence speeds and higher convergence accuracy but also significantly reduces instances of local optima entrapment. This leads to higher trajectory evaluation values compared to other swarm intelligence algorithms, verifying the effectiveness of IWOA in improving UAV swarm coordination and efficiency, even in complex and dynamic environments.

2 Whale optimization algorithm

Some hostile objectives or disaster areas must be reconnoitered, evaluated, or destroyed by UAVs, provided that we know the approximate locations of the targets. Afterward, the UAVs that have reached the destination will stay there temporarily to execute alert missions and send back surveillance information promptly. A homogeneous fleet of N UAVs takes off from a predetermined position X base station and searches the targets’ area defined in the continuous space. In view of the lack of GNSS signals in hostile or disaster environments, we generate a kite-inspired, connected UAV network step by step to locate the exact targets’ positions and execute the tasks. Once the UAVs find a group of targets with accurate information, the detection signal is sent to the base to arrange the following operation. The UAV continues to search the area until it must return to the base station for energy constraints.

(1) encircling prey:

In the behavior of encircling prey, individual whales choose to swim toward the optimal position or toward a randomly selected whale. This process resembles a cooperative behavior within the whale population, where others influence each whale, achieving better search and optimization effects through mutual collaboration. When an individual whale moves towards the position of the individual with the highest fitness in the population:

$$ \textstyle\begin{array}{l} \mathbf{D} = \left | \mathbf{CX}_{\mathrm{best}}(t) - \mathbf{X}(t) \right | \\ \mathbf{X}(t + 1) = \mathbf{X}_{\mathrm{best}}(t) + \mathbf{AD} \end{array} $$

(1)

where t represents the current number of iterations, $X_{best}$ represents the position of the optimal individual in the current population, X represents the position to be updated, A and C is the coefficient matrix, as shown in Equation (2):

$$ \left \{ \textstyle\begin{array}{c} \mathbf{A} = 2ar_{1} - a \\ \mathbf{C} = 2r_{2} \\ a = 2 - 2t/t_{\max} \end{array}\displaystyle \right . $$

(2)

where a is a convergence factor that decreases linearly during the iteration, $r_{1}$, $r_{2}$ is a random number between [0,1], and $t_{max}$ denotes the maximum number of iterations.

(2) bubble net attack:

Whales spray vapor packets to form bubble nets to repel prey while hunting. To do so, it needs to update its position constantly. The formula for updating the position of the whale when using a bubble net is as follows:

$$ \textstyle\begin{array}{l} \mathbf{X}(t + 1) = (\mathbf{D}')e^{bl}\cos (2\pi l) + \mathbf{X}^{*}(t) \\ \mathbf{D}' = \left | \mathbf{X}_{\mathrm{best}}(t) - \mathbf{X}(t) \right | \end{array} $$

(3)

It should be noted that, during the process of whales hunting prey, the selection of encircling and spiral bubbling methods is random. Probability settings for choosing the encircling mechanism and the spiral model, denoted as $P_{i}$ and $1 - P_{i}$ respectively, are employed to update the positions of whales. The mathematical model is represented as follows in Equation (4):

$$ \mathbf{X}(t + 1) = \left \{ \textstyle\begin{array}{c} \mathbf{X}^{*}(t) - \mathbf{AD}P_{i} \\ (\mathbf{D}')e^{bl}\cos (2\pi l) + \mathbf{X}^{*}(t)1 - P_{i} \end{array}\displaystyle \right . $$

(4)

where b is a constant whose value determines the shape of the logarithmic spiral, $\mathbf{D}'$ denotes the current distance between the whale and the prey, and b is a random number between [−1, 1].

(3) searching for prey:

During the search, humpback whales are randomized based on each other’s location. In the case that the information related to the prey has not yet been found (i.e., $\left | A \right | \ge 1$), an individual whale is randomly selected to update the position of other whales, increasing the randomness of the exploration in the hope of finding an even better position and strengthening the algorithm’s searching ability thus enabling the WOA algorithm to explore the globally optimal solution. In this case, the corresponding mathematical model is:

$$ \textstyle\begin{array}{l} \mathbf{X}(t + 1) = \mathbf{X}_{rand} - \mathbf{AD}_{rand} \\ \mathbf{D}_{rand} = \left | \mathbf{C} \oplus \mathbf{X}_{rand} - \mathbf{X}(t) \right | \end{array} $$

(5)

where $\mathbf{X}_{rand}$ denotes the position of a random individual in the current iteration.

In summary, the whale optimization algorithm, which determines the whale’s position update mechanism through the probability$P_{i}$, determines whether it is searching or exploiting by $\left | \mathbf{A} \right |$. The updating process is shown in Eq. (6):

$$ \mathbf{X}(t + 1) = \left \{ \textstyle\begin{array}{l@{\quad}l} \mathbf{X}_{\mathrm{best}}(t) - \mathbf{AD} & P_{i},\left | \mathbf{A} \right | < 1 \\ (\mathbf{D}')e^{bl}\cos (2\pi l) + \mathbf{X}^{*}(t) & 1 - P_{i} \\ \mathbf{X}_{rand} - \mathbf{AD}_{rand} & P_{i},\left | \mathbf{A} \right | \ge 1 \end{array}\displaystyle \right . $$

(6)

3 Improved whale optimization algorithm based on global optimization

While the Whale Optimization Algorithm (WOA) possesses advantages such as few parameters, simple structure, and fast convergence, it lacks operations to escape local optima. If it converges rapidly, there is a high likelihood of getting trapped in local optima. To address this issue, a method for global optimization, denoted as the Improved Whale Optimization Algorithm (IWOA), is proposed to enhance global search capability, convergence accuracy, and convergence speed. The optimized process of the Whale Optimization Algorithm (IWOA) is illustrated in Fig. 1.

Figure 1

Improved whale optimization algorithm flow

3.1 Description of the IWOA algorithm

(1) reverse learning

The distance between the initial and optimal solutions determines the speed and time of convergence of the WOA. The algorithm will converge faster if the initial is close to the optimal solution. Synchronizing a single solution and its inverse in each iteration increases the chance of approaching the optimal solution by at least half. It is then necessary to select the initial solution scientifically to be as close to the optimum as possible. Inverse learning can be used to improve the population intelligence optimization algorithm; the core idea is to compare the objective function values of the current solution and its inverse solution during the population optimization process and select a solution with a higher evaluation value as an individual in the next iteration. Elite reverse learning utilizes the characteristics of elite individuals carrying more effective information, constructs a reverse population to increase population diversity, and selects the optimal individual from the current population and the reverse population as the new individual enters the next iteration. The integration of an elite reverse learning strategy significantly enhances the algorithm’s convergence performance, diversity, and stability. Additionally, incorporating a stochastic weighting factor further improves the algorithm’s adaptability and overall effectiveness in solving optimization problems. Elite reverse learning is specifically defined as:

(a) Let $\mathop{x}\limits ^{\sim} = a + b - x$ be the reverse number of x, a real number $x \in \left [ a, b \right ]$.

(b) The elite solution is defined as the current high fitness solution by defining it as: $X_{i}^{*} = (x_{i}^{1*},x_{i}^{2*},x_{i}^{3*},\ldots,x_{i}^{d*})$, then the elite inverse solution can be defined as: $\mathop{X_{i}^{*}}\limits ^{\sim} = (x_{i}^{\mathop{1^{*}}\limits ^{\sim}},x_{i}^{\mathop{2^{*}}\limits ^{\sim}},x_{i}^{\mathop{3^{*}}\limits ^{\sim}},\ldots,x_{i}^{\mathop{d^{*}}\limits ^{\sim}} )$, $x_{i}^{\mathop{j^{*}}\limits ^{\sim}}$ defined as:

$$ x_{i}^{\mathop{j^{*}}\limits ^{\sim}} = k(ta_{j} + tb_{j}) - x_{i}^{j*} $$

(7)

where k takes random values in [0,1], representing the elite reverse coefficient. $x_{i}^{j*} \in [ta_{j},tb_{j}]$, $ta_{j} \in \min (x_{i}^{j})$, $tb_{j} \in \max (x_{i}^{j})$, $ta_{j}$, $tb_{j}$ is the dynamic boundary of the population. Since a fixed boundary cannot retain the experiential values of the search activities, a dynamic boundary allows the reverse solution to be constrained to a smaller search space, thereby accelerating algorithm convergence. If the reverse solution $x_{i}^{j*}$ exceeds the feasible solution boundary, a random value is selected within the dynamic boundary, and the reset method is represented as Equation (8):

$$ x_{i}^{j*} \in Rand(ta_{j},tb_{j}) $$

(8)

In summary, the whale population initialization process is as follows:

1) Randomly initialize a whale population and retain the top N/2 individuals with higher fitness to form the elite population.

2) Obtain the elite reverse population corresponding to the elite population.

3) Merge the search elite population with the elite reverse population, keeping the top N individuals with higher fitness as the initial population of whales.

(2) stochastic weighting factor

The WOA algorithm’s search behavior is influenced by the value of parameter A, which is itself related to changes in the convergence factor a. In the standard WOA algorithm, the value of a decreases linearly during iterations. This linear decrease can limit the algorithm’s adaptability and diversity, making it prone to getting trapped in local optima in the later stages of the search. To overcome this limitation, the Improved Whale Optimization Algorithm (IWOA) introduces a random convergence factor to enhance the algorithm’s exploration capability.

In IWOA, the convergence factor a is no longer controlled by a linear decrease over iterations. Instead, it is dynamically adjusted by randomly selecting its value during the iteration process. To ensure that the algorithm does not deviate excessively from optimal solutions, IWOA controls the variance of the convergence factor a, maintaining its randomness without reducing its exploratory potential. The algorithm increases its ability to explore diverse solutions and avoids premature convergence to local optima. The randomness of the convergence factor a in IWOA is implemented using the following Equation (9):

$$ a = Rand\left ( a_{\max} - a_{\min} \right ) + Rand\left ( \sigma N \right ) $$

(9)

where $a_{max}$ and $a_{min}$ represent bounds on the values of the convergence factor a, $Rand()$ is a function that produces standard normally distributed random numbers, and σ is the variance of the convergence factor of the random variable.

For the selection of these parameters, the following aspects should be considered:

1) The choice of boundaries directly affects the range of values that the parameter a can take, thus influencing the algorithm’s search scope. Choosing overly wide boundaries may lead to instability in the algorithm’s convergence, while too narrow boundaries may prevent the algorithm from effectively escaping local optima.

2) The variance σ determines the randomness of the parameter a. If the variance is too large, the randomness may be too strong, leading to ineffective convergence; if the variance is too small, it may not provide sufficient exploration capability. Therefore, the choice of σ should be adjusted based on the complexity of the specific problem.

3) By comparing the effects of random convergence factor settings in scenarios with varying complexity, the optimal parameter configurations for different situations can be determined. For example, simpler scenarios may not require high levels of randomness, while in more complex scenarios, moderately increasing the variance or expanding the boundaries may help avoid local optima [26].

(3) end stochastic neighborhood perturbation

The WOA algorithm may prematurely converge to suboptimal solutions due to insufficient initial population or a lack of exploration strategy. In this study, random perturbations are applied to the space near the solutions in the classical WOA, and random exploration is conducted in the vicinity of the better solutions to seek more optimal solutions. Random neighborhood disturbance exploration introduces population diversity, enhances algorithm convergence speed and time, and helps avoid local optima, making the algorithm more robust. The stochastic neighborhood perturbation is denoted as:

$$ \bar{\mathbf{X}}(t) = \mathbf{X}(t) + Rand( - 0.5,0.5) \times \mathbf{X}(t) $$

(10)

where $\bar{\mathbf{X}}(t)$ is a new solution after random perturbation, $\mathbf{X}(t)$ represents the current solution during the iteration, and $Rand( - 0.5,0.5)$ is a random number generation function evenly distributed between −0.5 and 0.5.

In IWOA, random neighborhood perturbations are introduced to increase the algorithm’s exploration ability and prevent it from falling into local optima prematurely. However, the choice of disturbance settings needs to be carefully considered, and the following factors should be taken into account when selecting disturbance parameters:

1) The magnitude of the perturbation should be commensurate with the size of the solution. A perturbation that is too small may not effectively help the algorithm escape from local optima, while a perturbation that is too large may cause the algorithm to lose direction, resulting in a decline in global search efficiency. To ensure that the perturbation is within an appropriate range, the amplitude of the perturbation is usually set as a proportion of the current solution, i.e., $Rand( - 0.5,0.5) \times \mathbf{X}(t)$.

2) Regarding the rationality of the disturbance settings, some formulas can be introduced to quantify and explain the effect of the disturbance. The disturbance amplitude can be expressed as:

$$ \bar{\mathbf{X}}(t) = \mathbf{X}(t) + \delta \times \mathbf{X}(t) $$

(11)

where δ is a disturbance factor, defined as: $\delta = Rand( - \lambda ,\lambda )$, λ is a positive number that represents the maximum amplitude of the disturbance, usually between 0 and 0.5.

To quantify the effect of the perturbation on the solution, the expected value $E(\delta )$ and the variance $\operatorname{Var}(\delta )$ of the perturbation amplitude can be defined to evaluate the perturbation effect:

$$\begin{aligned}& E(\delta ) = \frac{1}{2\lambda} \int _{ - \lambda}^{\lambda} \delta d\delta = 0 \end{aligned}$$

(12)

$$\begin{aligned}& \operatorname{Var}(\delta ) = \frac{1}{2\lambda} \int _{ - \lambda}^{\lambda} (\delta - E[\delta ])^{2} d\delta = \frac{\lambda ^{2}}{3} \end{aligned}$$

(13)

The variance $\operatorname{Var}(\delta )$ determines the intensity of the disturbance, and a larger λ value increases the intensity of the disturbance, thereby enhancing the exploration ability, while smaller λ values help maintain convergence stability.

3) Usually, in IWOA, the newly generated solution replaces the current solution if it is better; otherwise, the original solution is retained. This strategy balances exploration and exploitation with moderate selection pressure, preventing the algorithm from being too biased towards one local area. Adaptively adjusting this condition (for example, by setting a dynamic threshold for acceptance criteria) can further optimize the perturbation effect [27].

(4) algorithm analysis

The introduction of elite inversion increases the algorithm’s computational complexity, primarily due to the additional fitness calculations and individual location updates required. Each elite inversion operation necessitates evaluating the fitness of newly generated individuals, thereby increasing the computational workload for each generation. Moreover, these new individuals must be updated and selected, further adding to the algorithm’s computational burden. The time complexity of elite inversion is linear with respect to the population size and number of iterations.

Assuming a population size of N, fitness evaluations are required for each newly generated elite solution during the inversion process. If k elite individuals are generated per generation, the number of fitness calculations per generation increases by k. Additionally, updating the positions of these newly generated elite individuals also contributes to the linear increase in computational cost per generation.

Therefore, the additional computational cost of elite inversion can be expressed as:

$$ T_{\mathrm{elite}} = O(k \cdot N \cdot t_{\mathrm{fit}}) + O(k \cdot N \cdot t_{\mathrm{update}}) $$

(14)

where $t_{\mathrm{fit}}$ is the time of a fitness calculation, and $t_{\mathrm{update}}$ is the time of a position update.

Since k is usually much smaller than N, the time complexity of this part can be approximately reduced to $O(N)$.

Although the introduction of elite inversion increases computational costs, it significantly enhances population diversity, which is crucial for preventing the algorithm from getting stuck in local optima. This increased diversity allows the population to explore the solution space more thoroughly, thereby improving the chances of finding a globally optimal solution. In complex problems like path optimization, elite inversion often guides the algorithm toward better regions of the solution space, significantly improving the final solution quality.

Additionally, the algorithm’s randomness is increased by introducing the convergence factor a, which slightly raises the computational complexity. This randomness is typically introduced by selecting a random in each iteration, which can lead to a small increase in running time. Moreover, adding perturbations to the neighborhood space to enhance local search capabilities slightly increases the computational load per iteration.

Assuming that the random selection operation is performed once in each generation and has a computational cost of $O(1)$, the additional cost of this part is almost negligible. However, perturbing the solution space in each iteration increases the computational cost of local search. If the perturbation operation affects the update of each individual and requires additional neighborhood calculations, its additional computational cost is:

$$ T_{\mathrm{rand}} = O(N \cdot t_{\mathrm{disturb}}) $$

(15)

where $t_{\mathrm{disturb}}$ is the time of one perturbed operation. In general, the effect of introducing random factors on time complexity is also linear $O(N)$.

Introducing random factors primarily enhances the algorithm’s exploration ability, reducing the risk of falling into local optima. When optimizing complex multimodal functions, the added randomness allows the algorithm to explore the solution space more thoroughly, increasing the likelihood of finding a global optimum. Additionally, perturbations in the local space help the algorithm escape local optima more effectively, further improving solution accuracy.

Overall, the fitness calculation still largely determines the complexity of the improved IWOA algorithm. The various randomness and perturbation operations introduced increase the computational cost but are usually manageable. The complexity is related to the number of iterations, the population size, the adaptation calculation time, and the specific randomness parameter settings. In practice, the computational cost can be reduced by adjusting the parameters and reasonable computational optimization methods to achieve better algorithm performance.

3.2 Theoretical and simulation analysis of IWOA algorithm performance

In order to evaluate the performance of the IWOA algorithm, this study compares it with other existing algorithms, including the Grey Wolf Optimization (GWO) [21], Whale Optimization Algorithm (WOA) [16], Improved Grey Wolf Optimization (CGWO) [22], and Improved Whale Optimization Algorithm (GSWOA) [23]. The test instances utilize the CEC2014 standard functions, consisting of six benchmark functions, as presented in Table 1. Functions F1, F2, F3, and F4 are unimodal functions, possessing a global optimum but no local optima, making them suitable for evaluating the convergence ability of optimization strategies. In contrast, F5 and F6 are multimodal functions, where each peak corresponds to a local optimum, but there is only one global optimum. Due to the presence of multiple local optima, algorithms with insufficient search capability may easily get trapped in these local optima, failing to find the global optimum. Regarding algorithm development capability, as the dimensionality of functions increases, the optimization complexity also rises. Algorithms with weaker development capabilities may exhibit suboptimal performance when optimizing multimodal functions.

Table 1 Test Function

The above algorithms are tested on F1 F6 benchmark functions with the same parameters: the scale N is 30 individuals, the maximum number of iterations is 100, and each algorithm is run individually for 30 tests. The test statistics are shown in Table 2.

Table 2 Algorithm Run Time Comparison Results

The average values visually represent algorithm performance, while the variance indicates algorithm stability. According to the statistical results, for unimodal functions F1, F2, and F3, IWOA demonstrates higher precision in obtaining optimal solutions. When considering function F4, both IWOA and GSWOA consistently find the optimal solution in each statistical analysis, and the iteration count can be inferred through convergence analysis. In the case of multimodal functions F5 and F6, IWOA achieves the highest precision. Thus, across the testing scenarios of F1 to F6, IWOA outperforms existing algorithms.

Convergence speed and time are also crucial performance indicators that need to be examined. Figure 2 provides convergence test results for six benchmark functions.

Figure 2

Test function convergence analysis

From Fig. 2(a), (b), and (c), it can be observed that for functions F1, F2, and F3, the performance of several tested algorithms, including GWO, WOA, CGWO, and GSWOA, is relatively close. However, IWOA shows a faster convergence rate in the early stages and demonstrates greater exploration capabilities in the later stages, leading to high-precision solutions. This is primarily due to IWOA’s adaptive mechanism, which dynamically adjusts key parameters such as convergence factors, allowing the algorithm to focus more on exploration in the early stages and preventing premature convergence to local optima. In the later stages, IWOA gradually shifts to exploitation, effectively targeting promising regions within the solution space, thereby achieving higher precision.

From Fig. 2(d) for function F4, although both GSWOA and IWOA can obtain the optimal solution, IWOA has a significant advantage in terms of the number of iterations required. This advantage is due to IWOA’s refined exploitation mechanism in the later stages, which ensures not only fast convergence but also high precision. Additionally, IWOA’s population diversity maintenance strategy allows it to sustain population diversity even in later iterations, preventing premature convergence and stagnation, thereby further enhancing the algorithm’s flexibility and scalability.

From Fig. 2(e)(f), for functions F5 and F6, IWOA also demonstrates rapid convergence during the initial iterations. Compared with GWO, WOA, CGWO, and GSWOA, IWOA is particularly prominent in the later stages. The other algorithms tend to perform poorly in these stages and are prone to falling into local optima, which reduces the likelihood of finding globally optimal solutions—especially when dealing with complex multimodal problems—resulting in slower convergence. Through its enhanced search strategy and balanced exploration-exploitation mechanism, IWOA effectively avoids the local optima dilemma, maintains a certain degree of randomness and perturbation, and is capable of escaping local minima even in the later stages, continuing to search for the global optimal solution and achieving higher precision.

In contrast, GWO and CGWO may perform well initially, but they are more prone to falling into local optima in complex multimodal problems due to insufficient late-stage exploration capability. WOA and GSWOA improve on this to some extent, but they may still exhibit imprecise convergence when faced with highly complex functions.

A key factor in IWOA’s superior performance is its introduction of an opposition-based learning approach. This approach significantly enhances the algorithm’s exploration capability and maintains solution diversity throughout the iteration process. The opposition-based learning method expands the search space coverage by introducing opposing solutions in each generation, preventing the population from prematurely concentrating in a certain suboptimal region. This not only improves the algorithm’s global search capability but also enhances solution diversity during the iterative process. By continuously generating opposing candidate solutions, IWOA can more effectively escape local optima, thereby improving solution accuracy and enhancing the discovery of global optimal solutions in the later stages of convergence.

4 Test validation of uav swarm trajectory planning based on IWOA

A simulation testing environment is constructed to validate the algorithm’s effectiveness under real scenario constraints. UAV trajectory planning needs to find an optimal path to minimize the cost of UAV trajectory under the constraints of the UAV itself, spatial environment constraints, and other unexpected threats constraints. Various factors need to be considered when optimizing UAV trajectory planning. The shortest path that can avoid all threats cannot be simply regarded as optimal. On the one hand, to fly safely, the UAV has to fly within the area, while its flight altitude has to be restricted by the map and kept a certain safety distance from the ground. On the other hand, due to threat sources such as anti-aircraft artillery, it is necessary to use near-ground turns to avoid them. Finally, due to the limitation of carrying fuel, the UAV trajectory should be as short as possible to reach the destination as soon as possible. Combining the UAV state and environmental constraints, the test and validation flow of the UAV trajectory planning algorithm based on IWOA is shown in Fig. 3.

Figure 3

Test and validation process of UAV trajectory planning algorithm based on IWOA

4.1 Environment modeling

Environment modeling involves simulating structures such as buildings and mountain peaks. It serves as the foundation for UAV trajectory planning, and we employ a grid-based Digital Elevation Model (DEM) for environmental modeling. DEM, a Digital Terrain Model (DTM) branch, digitally simulates ground terrain using limited elevation data. It possesses various expressive capabilities, is consistently accurate, and offers real-time updating advantages. Typical forms of DEM, such as contour models, gridded DEM, and Triangulated Irregular Network (TIN), find extensive applications in UAV trajectory planning. DEM divides the planning area into grids of varying shapes depending on the axis orientation. Along the xy-plane, the grid size is set to 50-meter squares, which determines the accuracy and resolution of the terrain data in that plane. For the xz and yz planes, the grid is configured as 50-meter by 20-meter rectangles, providing a different level of detail and resolution along these axes. Mathematically, the terrain is expressed as a finite sequence of three-dimensional vectors on a region $V_{i} = (X_{i},Y_{i},Z_{i});i = 1,2, \ldots ,n$, where each vector corresponds to a specific grid point, capturing the elevation and spatial coordinates within the defined grid structure. This configuration allows for a more precise representation of terrain features while accounting for variations in elevation and other environmental factors. Here, $\left ( X_{i}, Y_{i} \right )$ represents the coordinates of specific terrain features mapped onto a plane, corresponding to the respective point’s elevation. The three-dimensional environmental terrain model can be obtained through Equation (16):

$$ H(x,y) = h_{0} + \sum _{i = 1}^{I} h_{i} \exp \left [ - \left ( \frac{x - x_{i}}{x_{si}} \right ) - \left ( \frac{y - y_{i}}{y_{si}} \right ) \right ] $$

(16)

where $h_{0}$ is the base ground height, $h_{i}$ is the height of the I-th mountain, $\left ( x_{i},y_{i} \right )$ is the coordinate of the center of the Ith mountain on the map, $x_{si}$ and $y_{si}$ are the horizontal slope and vertical slope of the Ith mountain, respectively.

Ensuring trajectory reliability is critical in studying static path planning for UAVs, which requires accurately identifying various threat factors, including electromagnetic interference, radar detection, and air defense artillery. These threat factors can significantly impact both UAV flight safety and mission execution. When performing static path planning, it is first necessary to thoroughly analyze the environment, including identifying potential sources of electromagnetic interference and radar coverage areas and determining the location and effective range of possible anti-aircraft artillery. This information helps establish a realistic threat model for accurate consideration in the path-planning process.

In this study, the experiment was conducted in a simulated mountainous environment characterized by significant terrain elevation changes and designed to test UAV path planning capabilities in complex terrain. The terrain features multiple peaks with a maximum elevation difference of 100 meters and slope angles ranging from 10° to 45°. The UAV’s flight height is maintained above 25 meters to avoid ground obstacles, ensuring that the UAV can quickly reach the target area while avoiding radar detection and mountain peaks.

Based on these considerations, two common threats, radar and anti-aircraft artillery, are simplified as cylinders in the model. The radar coverage area is represented by cylinders with a radius of 40 meters and a height of 90 meters. This simplification improves the trajectory planning model’s efficiency, allowing for easier simulation and calculation during the planning process while maintaining reasonable accuracy. In summary, the overall environment modeling, including the described terrain and threat scenarios, is illustrated in Fig. 4.

Figure 4

Environmental modeling

4.2 UAV performance constraints

In the static global path planning process for UAVs, treating the UAV as a point mass allows for model simplification, reducing computational complexity. The focus is placed on performance constraints such as minimum step size, pitch angle, and yaw angle. To enhance planning efficiency within this simplified model, UAV motion is simplified as a combination of straight lines and curves.

(1) minimum step size

In UAV trajectory planning, the minimum step size refers to the minimum distance the UAV moves between each step (or waypoint) in the planning process. The trajectory path L should not be less than the minimum step size. By appropriately setting the minimum step size, optimizing and smoothing the UAV flight path while maintaining trajectory planning accuracy is possible. This ensures that the UAV can execute tasks efficiently and stably.

(2) maximum yaw angle

The maximum yaw angle is a crucial constraint in UAV trajectory planning, limiting the UAV’s turning capability between each waypoint in the planning process. During flight, the UAV must maintain a stable and controllable state, avoiding excessively sharp turns that could lead to flight instability or loss of control. Therefore, appropriately restricting the maximum yaw angle helps ensure the safety of UAV flight, preventing situations of instability or lack of control during the flight process. The heading of the ith segment of the route is denoted as $a_{i}$, and the heading of the next segment is denoted as $a_{i + 1}$. The constraint on the maximum yaw angle, denoted as ψ, can be expressed as:

$$ \frac{\mathbf{a}_{i}\mathbf{a}_{i + 1}}{|\mathbf{a}_{i}| \bullet |\mathbf{a}_{i + 1}|} \ge \cos \psi $$

(17)

(3) maximum pitch angle

Pitch angle refers to the angle between the UAV flight direction and the horizontal plane. In trajectory planning, for different types of missions, the maximum pitch angle may be set differently, and it is necessary to reasonably limit the range of the UAV’s pitch angle according to the specific situation to ensure that the UAV always remains stable during flight. Let $\varphi _{max}$ be the maximum allowable climb angle or pitch angle, and the constraint is expressed as:

$$ \tan \left ( \frac{z_{i + 1} - z_{i}}{x_{i + 1} - x_{i}} \right ) \le \tan \varphi _{\max} $$

(18)

4.3 Comprehensive evaluation function

The quality of the planned path directly impacts the efficiency, safety, and mission execution capabilities of the UAV. To assess the quality of the planned path, a suitable objective function is established, considering the cost of fuel consumption, the cost associated with flight altitude, and the overall threat cost to the UAV. Various constraints influencing the path quality are also taken into account.

(1) path length (fuel consumption evaluation)

To simplify the model, the energy consumption of the UAV is defined using the path length, evaluated as shown in Equation (19):

$$ L = \mu \bullet \sum _{i = 1}^{k} \sqrt{\left ( x_{i + 1} - x_{i} \right )^{2} + \left ( y_{i + 1} - y_{i} \right )^{2} + \left ( z_{i + 1} - z_{i} \right )^{2}} $$

(19)

where μ is the distance coefficient, $\left ( x_{i}, y_{i}, z_{i} \right )$ and $\left ( x_{i + 1}, y_{i + 1}, z_{i + 1} \right )$ represent the coordinates of the ith and (i+1)th path point of the UAV, respectively, and k is the set of path points.

(2) altitude assessment

The flight altitude of the UAV can be evaluated by equation (20):

$$ H = \varpi \bullet \sum _{i = 1}^{k} h_{i} $$

(20)

where ϖ is the constant coefficient of flight altitude, $h_{i}$ denotes the value of the ith altitude sampling point, and k is the number of sampling point sets.

(3) threat assessment

Radar and artillery are the main facilities that threaten the safe flight of UAVs; in order to measure the degree of threat to UAVs, the threat cost model is designed here as follows:

$$ F = \sum _{j = 1}^{M_{1}} \partial _{j} \cdot (D_{d} - D_{j}) + \sum _{n = 1}^{M_{2}} \beta \cdot (D_{s} - D_{n}) $$

(21)

The threat-cost model quantifies the danger faced by UAVs, where $\partial _{j}$ is the threat coefficient of the jth threat source and $D_{j}$ is the distance to the jth source. β is the threat coefficient of UAVs in the adjacent airspace, $D_{n}$ is the distance to the nth UAV in the adjacent airspace, $D_{n}$ and $D_{s}$ are the desired safety distance, and $M_{1}$, $M_{2}$ is the total number of threat sources and the total number of UAVs in the adjacent airspace, respectively. In summary, the comprehensive assessment function is mainly composed of four parts, and the comprehensive assessment function is:

$$ J = k - (\lambda _{1} \cdot L + \lambda _{2} \cdot H + \lambda _{3} \cdot F)s.t.\lambda _{1} + \lambda _{2} + \lambda _{3} = 1 $$

(22)

where $\lambda _{1}$, $\lambda _{2}$, $\lambda _{3}$ is the weighting factor and k is the assessment value.

4.4 Testing and verification analysis

The simulation tests are conducted to validate the effectiveness of the actual trajectory planning based on the IWOA algorithm and analyze its strengths and weaknesses on a fixed map under the same hardware conditions. The flight environment model is tested using a grid map with dimensions of 200 m * 200 m * 100 m, where cones and cylinders represent peaks and obstacles, with coordinates as listed in Table 3. Three UAVs with starting coordinates (0,0,20), (10,0,20), and (0,10,20), while the target points were (200,100,40), (200,200,40), and (200,200,20). The test involves 100 iterations, 70 populations, and single-machine parameters, including a maximum turning angle limit of 90°, a maximum pitch angle limit of 50°, and a minimum step size of 10.

Table 3 Experimental Environment Map Data

The simulation results of path optimization by WOA, GSWOA, CGWO, and IWOA algorithms are depicted in Fig. 5. Constraints such as the flight map and the starting and ending points are the same for all algorithms. Each algorithm successfully navigates around peaks and obstacles, but they exhibit distinct optimal paths. The respective path distances are as follows: 1741.0118 meters for WOA, 1509.160 2 meters for GSWOA, 1647.2894 meters for CGWO, and 1099.7652 meters for IWOA. Notably, the IWOA algorithm ensures a faster convergence in the early stages of path optimization, and in the later stages, it surpasses local optima, resulting in a shorter overall path.

Figure 5

Search path and convergence analysis of each algorithm

For the comprehensive evaluation of paths, the following parameter is set to $\lambda _{1}$ = 0.5, $\lambda _{2}$ = 0.25, $\lambda _{3}$ = 0.25, k=100. A comparison of the path evaluation values is given in Fig. 6. IWOA converges fast in the first half of the process and has the highest comprehensive evaluation value. Each time the simulation is simulated, the stability of the algorithms is greatly affected by the initial solution variance. For example, the CGWO algorithm occasionally performs better but is not stable, and the poor variance of the optimization search results is large. Therefore, in order to eliminate the initial solution variance, the algorithms need to be run independently of each other 30 times, thus obtaining 30 statistical samples. Figure 7 shows the mean value of the path evaluation, and its standard deviation is shown in Fig. 8.

Figure 6

Path evaluation value

Figure 7

Path evaluation mean

Figure 8

Mean square error of path evaluation

By comparing the mean and variance of the evaluation values, the IWOA algorithm has a high mean, low variance, and more stable pathfinding. In conclusion, IWOA can rapidly identify a secure path, especially exhibiting good convergence in the initial stages of pathfinding, and it achieves a global optimum in the later stages.

5 Conclusion

In the field of trajectory planning for small-scale unmanned aerial vehicle (UAV) swarms, traditional intelligent algorithms and recently developed bio-inspired algorithms often struggle with local optima, resulting in suboptimal solutions. Additionally, these algorithms typically fall short of providing optimal solutions and maintaining convergence speed in high-speed dynamic UAV networking scenarios. To address these challenges, this paper introduces an Improved Whale Optimization Algorithm (IWOA) that incorporates a global perspective. To enhance the diversity of initial solutions, an opposition-based learning method is employed, which constructs a reverse population. This approach improves the initial population’s diversity. To prevent the algorithm from becoming trapped in local optima, random convergence factors and end-point random neighborhood perturbations are integrated, thereby accelerating the IWOA’s global convergence speed. Furthermore, a grid digital elevation model is utilized in the experimental environment model to account for various physical constraints of the UAV swarm, ensuring that the simulation validation of the IWOA algorithm closely mirrors real-world scenarios. Simulation results demonstrate that, compared to commonly used algorithms, the IWOA can generate more optimal trajectories for UAV swarm planning under conditions that resemble real-world constraints. From Figs. 6 to 8, it can be observed that the algorithm demonstrates improvements in trajectory evaluation, convergence speed, and stability under the tested conditions, indicating its potential to meet the trajectory planning requirements in small-scale UAV swarm scenarios involving threats and obstacles. The proposed IWOA significantly enhances UAV swarm coordination and efficiency, providing a robust solution to complex path planning challenges and underscoring its innovative contribution to the field of UAV optimization.