In this section, we test several examples to examine the effectiveness of the proposed scheme and compare the results with own exact solutions. Note that there are several minimization algorithms used in the neural network techniques. As mentioned above, we concentrate only on the efficiency of the proposed algorithm for solving ODE systems, without any aid of effects caused by other techniques such as the choice of neural networks, or minimization schemes, etc. Therefore, for these experiments, we simply use the basic gradient descent method as a minimization tool with the fixed learning rate \(\gamma = 0.001\). The details of each problem will be explained in each subsection.
All numerical results are obtained using Python 2.1.5, on a computer with 11th Gen INTEL Core I7-1165G7 CPU, 16 GB memory, and WIN10 operating system. All computational codes including neural networks and minimization schemes are implemented in Python by ourselves without any usage of libraries or packages. Also, since a simple neural network is used for this work, only 3 layers are used.
Example 1
For the first example, we test the simplest form of the ODE described by
$$ y'= \lambda y, \quad \lambda < 0, $$
(9)
with the initial condition \(y(0)=1\). The time interval is \([0,1] \), which is uniformly discretized into 10 subintervals. That is, 11 node points are used in the input and middle layers of the neural network. The analytic solution is \(y(t) = \exp (\lambda t)\). Note that to sustain the numerical stability, λ in (9) should be negative [1, 9, 14]. For this test, we simply set \(\lambda = -1\).
Since a provisional solution ŷ is estimated by the first-order explicit Euler method, the corresponding error has the second-order \(O(h^{2})\) magnitude, so the neural network term can be set up to be \(t^{2} N(t, w)\). The network was trained on the 10 subinterval points in \([0,1]\). All numerical results are plotted in Fig. 1(a) and compared with the analytic and provisional solutions. It can be seen that the proposed scheme is closer to the analytic solution, so we can conclude that the proposed scheme produces quite reasonable results.
For further examination of the accuracy, we check the difference between the analytic and proposed solutions and plot it in Fig. 1(b). To precisely compare the difference with the provisional solution, the \(L_{2}\)-norm is measured. The residual between the analytic solution and proposed scheme and between the analytic one and the provisional solution are 0.05277 and 0.15504, respectively. Summing up these results, we can easily see that the proposed scheme works well and has a good accuracy for this problem.
Additionally, to investigate the effect of the stiffness in neural network techniques, we apply the propose scheme to a problem with a stiff component, so λ is set to −10 in problem (9) and is required to be mildly stiff. Since it becomes stiff, the corresponding provisional solution should be obtained from an implicit method as long as the same step size is persistently used in the nonstiff case. Hence, the provisional solution at the ith time integration point can be obtained by the first-order implicit Euler method described by
$$ \hat{y}(t_{i}) = \hat{y}(t_{i-1}) + h f \bigl(t_{i}, \hat{y}(t_{i})\bigr), $$
(10)
where h is a step size and \(\hat{y}(t_{0}) = 100\). The analytic solution is represented as \(y(t) = \exp (-10t)\). With the provisional solution having second-order magnitude, the solution is taken as \(y(t) = \hat{y}(t) + t^{2} N(t, w)\). Other conditions for the neural network are the same as above. Figure 2(a) displays the solution behaviors of proposed schemes and comparisons with the analytic solution. Note that due to the stiff component, the solution improved rapidly. Here, the figure is plotted in the log-scale to observe the magnitude of the solution. It can show that the results from the proposed scheme are closer to the analytic solution. To precisely check the residual, we plot the error between the results from the proposed scheme and the analytic solution in Fig. 2(b). It shows that the proposed scheme works well even for the stiff problem. However, the results are not perfectly satisfactory and we need to consider other possibilities to improve the results for stiff problems. Actually, there are lots of components to control in the neural network techniques, such as several choices of the minimization technique, free parameters in each minimization, the number of the free parameters, etc.
Example 2
Next, we consider the following ODE:
$$ y'+ \biggl(t+\frac{1+3t^{2}}{1+t+t^{3}} \biggr)y=t^{3}+2t+ \frac{t^{2}(1+3t^{2})}{1+t+t^{3}}, $$
(11)
with the initial condition \(y(0)=1\) and the time interval \([0,1]\), with a uniform step size \(h=0.1\). Similarly to the previous example, 11 nodes are used in each layer. The exact solution is \(y(t) = \frac{\exp (-t^{2}/2)}{1 + t + t^{3}} + t^{2}\).
Similarly as above, a provisional solution ŷ is estimated by the first-order explicit midpoint method, so the corresponding error has second-order \(O(h^{2})\) magnitude and the error is set to \(t^{2} N(t, w)\). We plot the results in Fig. 3(a) and compare them with the analytic and provisional solutions. We easily check that the proposed scheme is quite close to the analytic solution. To examine the residual between the analytic solution and the results above, we plot each residual in Fig. 3(b). It can be concluded that the proposed scheme works well and has a good accuracy for this problem. Also, we check the computational time for the proposed scheme. Notice that for increasing the reliability of computational time, we take the average by executing this code 100 times. It needs 6.2 seconds CPU time and 617 iterations for minimization of the parameters w.
Next, to investigate the effect of the convergence magnitude in the proposed neural network technique, we try to use higher-order provisional solutions. First, a provisional solution ŷ is estimated by the second-order explicit midpoint method as described by
$$ \hat{y}(t_{i}) = \hat{y}(t_{i-1}) + h f (t_{i-1}+\frac{h}{2}, \hat{y}(t_{i-1})+ \frac{h}{2} f\bigl(t_{i-1}, \hat{y}(t_{i-1}) \bigr), $$
(12)
where h is a step size and \(\hat{y}(t_{0}) = y_{0}\). Therefore, the corresponding error is \(O(h^{3})\), so the neural solution can be defined as \(t^{3} N(t, w)\). The results concerning the accuracy at grid points are presented in Fig. 4(a) with comparisons of the analytic and provisional solutions. It can be seen that the proposed scheme is closer to the analytic solution.
Since a higher-order solution is much closer to the analytic solution, we plot the residual between the analytic solution and the proposed one in Fig. 4(b) to inspect in details the accuracy of the proposed scheme. One can see that the proposed higher-order scheme overall has a quite smaller error compared with the second-order convergence method, although there are a few parts which have a larger error.
Example 3
In this subsection, the following differential equation is considered:
$$ y' + \frac{1}{5}y = \exp \biggl(- \frac{1}{5}t \biggr)\cos (t), $$
(13)
with the initial condition \(y(0)=0\). The exact solutions is \(y(t) = \exp (-t/5) \sin (t)\). For the experiments, we uniformly use 10 node points from \([0, 2]\) in the input and middle layers.
As done above, we estimate a provisional solution ŷ by the first-order explicit Euler method, so the corresponding error can be \(O(h^{2})\). The trial solution is formed to be \(y(t) = \hat{y} + t^{2} N(t, w)\). With the same setting for the neural network scheme, we generate all numerical results and plot them in Fig. 5(a). Also the results are compared with the analytic and provisional solutions. It can be seen that the proposed scheme is closer to the analytic solution. To take a closer look at the accuracy, we calculate the difference between the analytic solution and the proposed one and plot it in Fig. 5(b). The \(L_{2}\)-norm of the difference between the results from the proposed scheme and the analytic solution is 0.09291, whereas the \(L_{2}\)-norm of the comparison result is 0.24264. Therefore, as seen in the previous examples, we can easily conclude that the proposed scheme has a good accuracy for this problem.
Also, we check the error accuracy versus computational time of the proposed scheme. Since the computational time depends on the iteration times in the minimization scheme, we measure computational times and error accuracy by varying the iteration times. For the 100 iteration times, the accuracy is 0.17968 and it needs 1.1 seconds, and for the 2000 iterations, the accuracy is 0.02997 and the computational time is 10.5 seconds. Therefore, one can easily conclude that the more iterations we use, the more accurate results are generated. To support this argument, we plot error behavior according to the iterations in Fig. 6.
Example 4
As the last example, we consider a simple Hamiltonian system known as a harmonic oscillator:
$$\begin{aligned} y_{1}' &= y_{2}, \end{aligned}$$
(14)
$$\begin{aligned} y_{2}' &= -y_{1}, \end{aligned}$$
(15)
with the initial condition \(y(0)=[0,1]\). The exact solutions is \([y_{1}(t), y_{2}(t)] = [\sin{t}, \cos{t}]\). For the experiments, we uniformly use 10 node points from \([0, 2]\) in the input and middle layers.
Since the given system is Hamiltonian, a provisional solution ŷ is estimated by the first-order implicit Euler method described in Eq. (10), due to its stability. The trial solutions are set to \(y_{1}(t) = \hat{y_{1}} + t^{2} N_{1}(t, w)\) and \(y_{2}(t) = \hat{y_{2}} + t^{2} N_{2}(t, w)\). We simulate this vector system and generate the numerical result as seen in Fig. 7. The results are compared with the analytic and provisional solutions. It can be seen that both numerical solutions of the system have a quite good accuracy.
Additionally, a Hamiltonian system is a dynamical system described by the scalar function H, called the Hamiltonian. In this problem, the Hamiltonian H can be defined as
$$ H = y_{1}^{2} + y_{2}^{2}, $$
(16)
and the value of H should be conserved. To examine the conservation property, we plot an orbit of solutions in Fig. 8(a) and the Hamiltonian H in Fig. 8(b). Figure 8(a) shows that the proposed scheme is closer to the analytic solution. Also, one can verify in Fig. 8(b) that the implicit Euler method reduces the total energy of H, whereas results obtained from the proposed scheme try to conserve the energy after a certain moment.