https://doi.org/10.37815/rt= e.v33n2.852

Artículos originales=

Artificial intelligence-controlled pole balancing using an Arduino board<= /o:p>

Balanceo de un poste, controlado por una inteligencia artificial usa= ndo una placa Arduino

José Revelo¹ ^{https://orcid.org/0000-0003-1159-1523}, <= /span>Oscar Chang¹^{https://orcid.org/0000-0003-1241-1782}

¹Yachay, Imbabura, Ecuador

jose.revelo@yachaytech.edu.ec, oscar.chang@yachaytech.edu.ec

Sent: 2021/07/1= 1

Accepted: 2021/09/28

Published: 2021/11/30

Resumen

3D"Cuadro

El proceso de automatización (= AP) actual es de gran importancia en el mundo digitalizado, en rasgos generales, representa un aumento en la calidad de producción con el trabajo hecho a ma= no. El equilibrio es una capacidad natural del ser humano que está relacionada = en trabajos y conducta inteligente. Equilibrarse representa un desafío adicion= al en los procesos de automatización, debido a la presencia de múltiples varia= bles involucradas. Este artículo presenta el equilibrio físico y dinámico de un poste en el que un agente, mediante el uso de aprendizaje por refuerzo (RL), tiene la capacidad de explorar su entorno, detectar su posición a través de sensores, aprendiendo por sí mismo cómo mantener un poste equilibrado bajo perturbaciones en el mundo real. El agente usa los principios de RL para explorar y aprender nuevas posiciones y correcciones que conducen a recompe= nsas más significativas en términos de equilibrio del poste. Mediante el uso de = una matriz Q, el agente explora las condiciones futuras y adquiere información = de política que hace posible mantener el equilibrio. Todo el proceso de entrenamiento y pruebas se realizan y gestionan íntegramente en un microcontrolador Arduino. Con la ayuda de sensores, servo motores, comunicaciones inalámbricas e inteligencia artificial, todos estos componen= tes se fusionan en un sistema que recupera constantemente el equilibrio bajo cambios aleatorios de posición. Los resultados obtenidos demuestran que a t= ravés de RL un agente puede aprender por sí mismo a utilizar sensores, actuadores genéricos y resolver problemas de balanceo incluso bajo las limitaciones que presenta un microcontrolador.

Palabras clave: aprendizaje por refuerzo, agente inteligente, microcontrolador, Industria 4.0.

Abstract

Automation Process (AP) is an important issue in the current digitiz= ed world and, in general, represents an increase in the quality of productivity when compared with manual control. Balance is a natural human capacity as it relates to complex operations and intelligence. Balance Control presents an extra challenge in automation processes, due to the many variables that may= be involved. This work presents a phy= sical balancing pole where a Reinforcement Learning (RL) agent can explore the environment, sense its position through accelerometers, and wirelessly communicate and eventually learns by itself how to keep the pole balanced u= nder noise disturbance. The agent uses RL principles to explore and learn new po= sitions and corrections that lead toward more significant rewards in terms of pole equilibrium. By using a Q-matrix, the agent explores future conditions and acquires policy information that makes it possible to maintain stability. An Arduino microcontroller processes all training and testing. With the help of sensors, servo motors, wireless communications, and artificial intelligence, components merge into a system that consistently recovers equilibrium under random position changes. The obtained results prove that through RL, an age= nt can learn by itself to use generic sensors, actuators and solve balancing problems even under the limitations that a microcontroller presents<= span lang=3DEN-AU style=3D'mso-ansi-language:EN-AU'>.

Keywords: reinforcement learning, intelligent agent, Q-learning, microcontroller, Industry 4.0.

Introduction<= /span>

F= actories continue to improve and introduce new technology to reduce their production costs, such as the solution given to repetitive work in the delivery of goo= ds industry (Azadeh = et al., 2019). New technologies have to t= arget and solve two main aspects to keep progressing and improve tasks in multiple areas: transparency of costs that makes replication easier and produce these technologies; recreate a more human-like robot behavior.<= /p>

<= o:p>

O= ne primary aspect and basic act that robots should replicate from humans, is to be proficient at balancing. Whether we want a humanoid or a computer to replace the workers, they have to be capable of performing on a similar level to hu= mans (Hyon et= al., 2007). Ships’ displacement is oft= en dependent on careful balancing; in sea navigation, equilibrium is critical.= The balancing mechanism is based on fin stabilizers, which rotate and change the fin angle to resist ocean disturbance (Sun et = al., 2018). Balance is also present in aviation. Unmanned aerial vehicles (UAVs) use stabilizers that play an important role minimizing disruption effects and ensuring a smoother flight= (Korkmaz= et al., 2013), by dealing with three diff= erent angles known as yaw, roll, and pitch. Balancing is a critical component that affects the ability to perform a wide variety of tasks and is essential in = the operation require by loaders, robotic arms, ships, anti-seismic systems, or robots themselves.

<= o:p>

M= achine activity does not present adaptation or reaction to new events that occur, = and are not predictable during normal function. This is critical if correct balancing is a goal in changing environments. Unpredictable events and vary= ing conditions of the environment are why automation has limited uses in many factories and moving vehicles. AI may be set to perform a task in varying circumstances; it may give improved results, and be efficient in activities currently executed only by humans. A contemporary and relevant tool in AI is Reinforcement Learning (RL), a methodology by which agents independently le= arn efficient control policies. They then use this knowledge to solve logical or mechanical problems (Foerste= r et al., 2017).

<= o:p>

T= he first reason to use Arduino is its flexibility to adapt to different projects. AI requires data recollection and this may be a difficult task because data is proprietary or cost prohibitive. In this sense, Arduino is handy and very helpful for recollecting data from the world.

<= o:p>

A= rduino is reasonably priced for building several prototypes. Also, as shown in (López-R= odríguez & Cuesta, 2016; Taha & Marhoon, 2018), is how feasible it is to b= uild more than one specific model or prototype with customizable devices, depend= ing on the situation. The low cost advantage also decreases barriers to perform= a study in different areas of AI devices, or reducing problems to replicate a= nd produce multiple similar technologies that instantly interact with the real world.

<= o:p>

W= ith these premises, this paper develops a dynamic balancing pole that AI manages runn= ing in an Arduino microcontroller. Here, an RL agent, with the capacity to read sensors and control the angles of servo motors that drive an x, y balancing pole, learns by itself how to keep the pole balanced under noise disturbanc= e, responding to natural changes of its environment. The prototype demonstrates the capabilities of microcontrollers that have some limitations, compared w= ith regular desk computers, and how extensive is there potential.

Related work

T= he following section examines and evaluates works that are relevant to the pla= nted objectives and share common themes, strategies, and method elaboration.

<= o:p>

Arduino fully managed systems

D= ue to their low and medium cost, Arduino-based projects can be produced or recrea= ted. Due to easy access to a variety of hardware resources, it is simple to exec= ute multi-purpose projects. A mobile robotic platform is presented in ADDIN CSL_CITATION {"citationItems":[{= "id":"ITEM-1","itemData":{"author":= [{"dropping-particle":"","family":"Araúj= o","given":"André","non-dropping-particle&quo= t;:"","parse-names":false,"suffix":"&quo= t;},{"dropping-particle":"","family":"Po= rtugal","given":"David","non-dropping-particl= e":"","parse-names":false,"suffix":"= ;"},{"dropping-particle":"","family":&qu= ot;Couceiro","given":"Micael S","non-dropping-particle":"","parse-names&qu= ot;:false,"suffix":""},{"dropping-particle":&= quot;","family":"Rocha","given":"Rui P","non-dropping-particle":"","parse-names&qu= ot;:false,"suffix":""}],"container-title":&qu= ot;Journal of Intelligent \\& Robotic Systems","id":"ITEM-1","issue":"2&q= uot;,"issued":{"date-parts":[["2015"]]},"= ;page":"281-298","publisher":"Springer",= "title":"Integrating Arduino-based educational mobile robots in ROS","type":"= ;article-journal","volume":"77"},"uris":= ["http://www.mendeley.com/documents/?uuid=3D13d04b91-18e9-4b20-94af-8a= 79eeca6626"]}],"mendeley":{"formattedCitation":&qu= ot;(Araújo et al., 2015)","plainTextFormattedCitation":"(Araújo et al., 2015)","previouslyFormattedCitation":"(Araújo et a= l., 2015)"},"properties":{"noteIndex":0},"schema&= quot;:"https://github.com/citation-style-language/schema/raw/master/cs= l-citation.json"}(Araújo = et al., 2015). In the work, it is claimed= that robots must be as inexpensive as possible for students and researchers to conduct real-world experiments. The work in (Ram et = al., 2017) demonstrates the versatilit= y of Arduino when used in an IoT project, a proper power supply, wireless communication and sensor devices are good features of working with Arduino boards. The Arduino microcontroller is used in (Gonzále= z & Calderón, 2019) to build a Supervisory Cont= rol and Data Acquisition (SCADA) system with high configuration possibilities, easy access to libraries and a fast lea= rning rate for new users. In (Wu et a= l., 2017) it is described the Arduino= board's data collection, tracking, and scalability projection capabilities.

<= span style=3D'mso-spacerun:yes'>

T= he majority of the work discuss the low-cost advantages of Arduino projects, as well as how to adapt or include more electronic devices to create more robu= st devices with no issues. Furthermore, the majority of the works incorporate = two elements: automation and real-time operations. Arduino can conduct real-time operations without the use of simulations, concentrating on the problems th= at are important to the target, enabling it to detect planning mistakes or mis= sed steps in the early stages of development. This has been a big help for researches, both technically and theoretically, showing that Arduino is a v= ery useful tool.

<= o:p>

Artificial intelligence with microcont= rollers

I= n (Jain, 2= 018), a self-driving car is demo= nstrated using a Raspberry Pi and a Convolutional Neural Network. An Arduino board guides the movements of the vehicle. It receives an order and drives the ca= r in a specific direction. The author concludes that the car's design and testing were successful. In addition, the work (López-R= odríguez & Cuesta, 2016) shows a mobile robot for educational purposes based on Android and Arduino. The robot has a light sensor, GPS, camera, accelerometer, Bluetooth, and Wi-Fi, and can accept commands from a smartphone via an Internet connection. The author claims th= at software alteration can be easily applied for future projects. In the third study (Lengare= & Rane, 2015), image processing is used to monitor the movements of a human arm and reproduce this action in a robotic= arm controlled by an Arduino. The author claims that an Arduino board can monit= or the robot's behavior; however, there are many ways to accomplish the same t= ask.

<= o:p>

I= t is clear from the previous work that the Arduino will perform actions with good resu= lts if the instructions come from a logical process of an efficient AI or a hum= an. The argument is that with the use of a microcontroller, the majority of the work present no hardware problems. As a consequence, with the right instructions, Arduino will perform a variety of tasks. The intelligence of = data analysis is executed outside the microcontroller in all previous projects, = but as further projects illustrate, it is also possible to code the intelligenc= e or incorporate learning techniques in the Arduino itself, without having to ra= ise costs or resources.

<= o:p>

A= rtificial Intelligence has shown better results than conventional approaches in sever= al of projects; these works involve intelligent agents that perform several ta= sks in pursuit of optimum results. It was shown in (Jimenez= et al., 2020; Salazar et al., 2013), that an intelligent agent’s capable of considering timing, amount of water, and properly implementing t= hem in what they detail as Spatio-temporal variations of the soil–plant–atmosph= ere system, gathering impressive results in terms of water efficiency and preci= sion irrigation. As agents work with microcontrollers like Arduino, they may cre= ate further applications. For example, (Mata-Ri= vera et al., 2019) shows an intelligent traffi= c light control device that uses Arduino sensors to detect the color of traffic lig= hts, as well as the tone, distance, and motion of vehicles in the streets where = it is installed. The entire process is executed in Arduino, demonstrating how = this platform can perform complex tasks using machine learning techniques.<= /o:p>

<= o:p>

S= ince the recently mentioned works used an Arduino to carry out the whole operation, these project reinforces the concept that Arduino hardware and software are plenty capable, and assisting this study in realizing that AI can be implemented in a microcontroller. Of course, Arduino has limitations, but t= he previous works only require a little extra effort to comprehend the obstacl= es, adapt the prototype, and include RL.

<= o:p>

Reinforcement learning in Real environ= ments

T= he aim of developing robots is to allow them to participate in a real-world environme= nt where unexpected events can occur. The goal is to make machines more effect= ive in performing various tasks in a manner that is equivalent to, if not super= ior to, a human. To accomplish this action in computers, a variety of techniques are used.

<= o:p>

T= hree classifications of methods are given in (Kormush= ev et al., 2013), as well as mentioned how c= omputers can learn to deal with such tasks. Direct programming is defined as the low= est method. This is more akin to a programming method than it is to learning. T= he second method is imitation learning, which is more comparable to a learning strategy. The problem with this approach is that it requires a specialist to perform a perfect presentation of how to complete the job. The last method,= RL is defined as a trial-and-error method that explores the environment and the robot's body. RL has three benefits over the other two approaches: learning= new tasks, optimizing efficiency, and adapting to new situations.

<= span style=3D'mso-spacerun:yes'>

R= L has been used in real-world settings to solve physical problems in various experimen= ts, but simulations can also be used in training. According to (Pan et = al., 2017), depending on the context, = the training will present undesirable driving behaviors that can cause harm to the environment. As a result, depending on the case, a simulation or real-world environment may be used to train a computer. The problem with transitioning from a virtual to a real-world environment is that it takes extra time to a= pply the learned skills to a different scenario than the one in which they were educated. The research of (Miglino= et al., 1995) demonstrates how simulation= s can solve problems that appear to be closer to the original target but are not;= in other words, the simulation environment lacks realistic behavior. However, conducting training in the real world poses a challenge; as mentioned in (Nagaban= di et al., 2018), samples can be extremely c= ostly. The tools used can influence this complication about cost of collecting samples. As mentioned, an Arduino’s main advantage is its low cost.

<= span style=3D'mso-spacerun:yes'>

T= he work from (Gu et a= l., 2017) creates and trains a robot = to open a door from scratch, which is a realistic application that requires complex contact dynamics to imitate human physical abilities. In addition, it demonstrates how real robotic platforms can perform physical tasks in 3D re= al environments. It is also claimed that RL is an effective training method for these systems. The work of (Sharma = et al., 2020) uses a free-reward RL algor= ithm to teach a robot how to travel properly within its structure and navigate, 20 hours of preparation, the machine had mastered a variety of locomotion gait= s. After a short time, the RL methods yielded results. This research showed ho= w RL can be used to teach an AI how to control its movements in the real world, without wasting too much time in the training process.

&n= bsp;

Methodology

T= his section describes the proposed system, which learns to detect a properly st= able position. The construction of the balancing prototype has three main points= to be described, divided into the following sections: General description of t= he system. A detailed description of the transmitter device. A detailed description of the receiver device. In the end, generating an algorithmic description of the training process.

<= o:p>

Problem Solving Phases

Problem Description

The structure to balance consists of a pole with a platform at the t= op, where objects can be placed. The pole cannot stand up straight due to a flexible component in the base, the pole is connected to this component whi= ch allows the pole to constantly bend in any direction. To control the pole displacement, two servo motors are connected using a rigid wire. One servo motor moves the pole up and down (north-south), the second mo= ves the pole from left to right (east-west). An upper view of the balancing structu= re shown in Figure 1 A, and a front= view in Figure 1 B.

Figure <= /span>1<= /span>

A View of the Structure to Balanc= e

3D"Interfaz

Analysis of the Problem

The first issue is to translate the environment in terms that is possible to convert into a programming platform, select the correct data ty= pe, and attempt not to exceed the memory capacity of the Arduino.

The speed execution of the code is enough to sustain the speed movem= ent of the servo motors. To train an agent and measure a reward, is necessary to measure after a change or move is completed. The Arduino board can execute multiple moves virtually before a servo motor actually has finished the fir= st move. In other words, the measurement has some imprecision issues, due to movements that are supposed to happen but are retarded because of inertial forces.

Implementation

The agent’s work is to randomly explore the environment through its = own actions. The agent will try different moves depending of its actual state, meaning that the agent will increase and decrease the angle in the servo, allowing the agent to explore which action will yield better or worse resul= ts, depending on the detected state. The Q-matrix, also known as a Q-table, represent the conditions and possible behavior of the agent represented. The dimensions of this Q-matrix are 9x2, represented in , where = the number of columns repres= ents the number of possible actions from which the pole will pass. The number of fil= es represents the inclination states in which the platform will be.=

Figure <= /span>2<= /span>

Q-Matrix, States(Rows) and Actions (Columns)

T= he se= rvo motors in use operate in degrees from 0⁰-180; one pushes the pole fo= rward or backward Figure 3 (a), while the= other moves the pole left Figure 3 (c), or right = Figure 3 (d). However, = if the system detects that is in balance, both servo motors will not move pole Figure 3 (b) and Figure 3 (e). The first= column depicts the action o= f decreasing the current angle at which the servo motor is located, while the second col= umn depicts the action of raising the current angle at which the servo motor is located. The servo motor has 144⁰ of inclination since the eighth row= is 8, which is the highest value that can be found in this matrix beginning at state 0 from 0⁰, in the servo motor.

<= o:p>

T= he error to balance the pole is 18⁰ due to the Q-matrix design. Both servos increase or decrease its angle 18⁰, the angle was chosen to save space memory in Arduino, and also the servo motors have a better response with an= gle variations of a larger magnitude, one-degree variation will be a good improvement, but better precision servo motors are required.

Figure <= /span>3<= /span>

Possible actions performed by the= servos

3D"Gráfico,

<= o:p>

T= he prototype stabilizes the post in 18 steps, with each servo performing nine steps simultaneously, in a worst-case scenario. Delays measured in the serv= o's movement speed, allowing the balancing time to increase or decrease. Servos perform angle variations every 200 milliseconds; hence these nine moves can= be completed in less than 1800 milliseconds. In the worst-case scenario, the prototype's balancing time is nearly two seconds.

<= o:p>

System description

The prototype design is structured by two devices working together to balance the pole, as seen in Figure 4. This details which componen= t is connected and how it communicates and flows. It is known as a Q-table, from upper components to which are below. the device A and B have wireless communication, the implementation and a deeper description of the internal process of each device will be described in the following sections. Device = A, due to its structure, could be reused in more areas to estimate more precise position changes. The device B does not have the same reuse capacity; it ha= s to be modified depending on the balanced structure.

Figure <= /span>4<= /span>

Prototype Whole Process Flow Diag= ram

Device = A

Has the task to recollect and notify positi= on changes of the platform in which it is placed. Figure 5 shows the electronic elements used as the Arduino Nano, the accelerometer and gyroscope that are in the MPU6050 senso= r, and the nRF24L01 RF transceiver. Device A performs the flow process in Figure 6.

Figure <= /span>5<= /span>

Two Pictures From a Different Angle, of Each Device and its Elements

Device B

This device, managed by Arduino Uno, will execute the process from <= /span>Figure 7. Afterwards, the Train process finishes and will execute the steps from Figure 8. Device A in Fi= gure 5 shows the two servo motors, that manage = the positions of the pole, also have the nRF24L01 RF transceiver for wireless communication.

Figure <= /span>6<= /span>

Arduino NANO Process Flow Diagram

3D"Diagrama

Descripción <= /o:p>

Device A process

Setup and Loop Functions

The setup and loop functions are the two main functions of most Ardu= ino programs. After uploading the program to an Arduino microcontroller, the se= tup function is the first to run. The loop function is a loop that repeats the actions inside the microcontroller until the microcontroller resets or the power supply is disconnected. Both functions are required in Arduino progra= ms.

Sense and Transmission Flow Process

The Arduino Nano carries out the entire process depicted in <= !--[if supportFields]> = REF _Ref87556241 \h Figure 6. First, is the setup function, where all variables of the accelerometer and gyroscope in the three-axis x, y, and z = are initialized in the variable initialization step. The data rate in symbols p= er second (baud) has several bits per second (bps) in which the Arduino will communicate during the transmission of data is setup. Sensor initialization begins the MPU6050 functionality after ensuring that it is operational. The Power Amplifier level, the channel between 2.400 and 2.524 GHz, the data ra= te, and the address of the receiver to which we will send data, are all set up = in the wireless communication initialization step.

<= o:p>

T= he loop, as seen in Figure 6, is the second part that constantly reads position changes. MPU6050 is constantly read during sensor reading, and the data obtained is then sent to data processing, where all information is processed in degrees and g (terms of standard gravity). After that, data transmission begins, which reports the changes in the pole's extreme to the Arduino UNO.<= /p>

<= o:p>

Device B process

Train Flow Process

Setup in a= nd a= re very similar, except for the variables initialization step with the R and Q matr= ices for the training process. As seen in Figure 6, use the same parameters for wireless communication initialization.

In the loop section, the data reception is emitted from the accelerom= eter in the Arduino Nano. Action Selection consists of a random selection of the available options from the servo motors. Now Perform action executes the selected action from the previous step. Calculating a reward is the next st= ep, updating the Q-table. A fixed number of iterations is achieved all the steps are repeated. The end section is saving the experience that is in the Q-tab= le, earned in the loop section for forthcoming testing.

<= o:p>

Figure 7<= /span>

Arduino UNO Train Phase Process F= low Diagram

3D"Diagrama

Descripción

Figure 8<= /span>

Arduino UNO Already Trained Process Flow Diagram

Testing Flow Process

Setup in a= nd s= how the same steps. Then in the loop function, the data reception is received from = the Arduino Nano, then it checks the Q-table, selects the best option, and perf= orms the action selected in the servo motors establishing the balance and repeat= ing the process from the data reception.

Code structure

T= he pseudocode from Figure 9 is train agent and executed to = train the agent. The input of the algorithm are the variables that describe the current state of the environment.

<= o:p>

F= irst are initialized the different variables the Qm is Q-matrix and all the values are initialized from zero, which will present changes in the values while learns the policy, K is the number of limited episodes, Moves is the variable that limits the number of moves an agent = can execute before finish an episode.

<= o:p>

A= fter the train starts, first are selected an x and y position, a random number selected between the 0 and the number of columns for x and the number of files for y of the Qm matrix, the variable = Agent is assigned a random starting action from Qm[x][y] position.

<= o:p>

T= he following inner loop works as a limited sequence of steps where the agent c= an move around the environment until it reaches the goal or executes a limited number of moves defined previously in the variable Moves. The function StepSelection looks at matrix Qm neighborhood values, selects an action of the possible options. The possible options of the act allow the a= gent to move in four possible ways; up, down, left, right.

<= o:p>

U= pdating the variables Agent, x, y, depe= nd on the previous move. The variable State saves the new environment variables. = The matrix Qm updates the index x, = y value calling the function rewar= d, which uses the variables State = and action to calculate the new value function Reward.

<= o:p>

I= f the agent reaches the goal state the episode finishes, otherwise this inner loo= p is repeated until the end of the allowed number of steps. The output is a Q-ta= ble matrix, the Qm which has the experience obtained after the agent exploration.

<= o:p>

Figure 9<= /span>

Agent Training Pseudocode =

3D"Interfaz

Results

Table = 1 shows the Q-table values for the servo mot= or that pushes the pole left and right, while Table 2 shows the Q-table values fo= r the second servo motor's forward and backward. The table depicts the Q-table du= ring various training episodes, and it is possible to observe how the absolute v= alue rises. The 0 value does not change during training because the episode ends when it enters this state; this state is considered the most stable because= it receives a reward of 0, while the other states receive a punishment dependi= ng on the platform's inclination. It is easy to see how values that are farther from the stable state are punished more severely, resulting in extreme valu= es gaining more knowledge, or learning faster than states closer to the optimal state. As compared to Q-tables with more iterations, it is possible to see = that certain values in one co= lumn that should be lower than the other column are not. In the first 50 episode= s in the table Q-table from Table = 1, it was determined that the agent does not have the experience to choose the proper actions at this point in the train= ing.

Table 1=

Agent 1 Q-table Evolution of Values at Different Training Episodes =

EPISODE	100	500<= /b>=	1000
State<= /o:p>	Actions
0	-7.63	-7.63	-16.42	-16.42	-16.13	-16.13
1	-7.59	-5.66	-17.45	-13.32	-17.44	-13.32
2	-3.91	-3.06	-12.95	-9.45	-12.96	-8.38
3	-1.44	-1.76	-6.56	-4.94	-8.48	-3.93
4	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>
5	-0.86	-0.54	-1.00	-2.33	-1.00	-2.75
6	-3.10	-4.99	-3.90	-7.04	-3.90	-6.77
7	-7.04	-9.41	-8.51	-13.99	-8.51	-12.37
8	-9.49	-9.49	-12.66	-12.66	-12.66	-12.66

Table 2Agent 2 Q-table Evol= ution of Values at Different Training Episodes

EPISODE=	50	100<= /b>=	500	1000
State<= /o:p>	Actions
0	-3.254	-3.54	-3.254	-3.54	-9.02	-9.02	-14.30	-14.30
1	-2.15	-1.73	-17.45	-1.73	-10.21	-8.22	-15.15	-13.49
2	-3.35	-2.24	-12.95	-2.24	-9.13	-7.74	-13.52	-10.74
3	-3.75	-2.77	-6.56	-2.77	-9.04	-3.99	-11.75	-6.32
4	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>	0<= /p>
5	-0.83	-0.47	-0.83	-0.47	-1.00	-1.46	-1.00	-1.67
6	-1.18	-1.73	-1.18	-1.73	-2.68	-3.38	-1.90	-3.40
7	-2.68	-4.69	-2.68	-4.69	-3.71	-7.44	-3.71	-6.50
8	-5.78	-5.78	-5.78	-5.78	-7.34	-7.34	-7.34	-7.34

In the cumulative reward Figure 10, the x-axis represents the number of episodes trained, while the y-values represent the absolute value total of each variable corresponding to the Q-table. The gra= phic demonstrates that the agent is learning, gaining experience after each iteration from the first to the nth episode. Figure 10 plots a summar= y of all the learning work, both curves present a general growth doing decently. Over the last two hundred iterations agent 1 does not have a noticeable increase in compari= son to earlier episodes. Agent 2 shows that it could improve its experience, th= is behavior comes from the random aspect of the training process, at some epis= odes would have a noticeable increase in comparison to earlier episodes, this behavior comes from the random aspect of the training process, at some epis= odes will have a noticeable.

Figure 10

Cumulative Rew= ard as a Function of the Number of Episodes, from both Agents

3D"Título:

In Figure 11 are the Q-matrix at episode 100 and 1000, = of both agents. The colors with a more negative value represent a higher point= and have a darker color, while values closer to zero are more clear and lower points. The agent's path will be the clearest option where it w= ould be located, at episode 1000, the disparity in colors between the different states of the Q-matrix is more apparent compared at episode 100, where the = path to follow is somewhat diffuse.

Figure 11

Agent 1 and Agent 2 Heatmap Q-matrix representation

= The pole is at the leftmost position; Figure 12 A depicts the direction in wh= ich the servo will move the pole; internally, the agent determines which alternativ= e is most appropriate. Figure 12 B is a basic example that con= cludes when the pole is balanced.

Figure 12

Prototype balancing the pole simple case = =

Figure 13 shows an interesting behavi= or, which an external disturbance is manually introduced, tilting the base, sim= ilar to what happens in air and sea ships. The agents must now solve a fresh and unexpected problem.

In Figure 13 A, B, C, the agent 2, tilte= d back, is in charge of balancing the pole in a forward-backward motion. Agent 2 be= gins to increase the servo's angle by 18 degrees until it is balanced in this direction; nevertheless, this movement has influenced the pole's correct position in the left-right direction; as a result, agent 1 begins to tilt t= he pole to the left, Figure 13 = D, E, and F depict the proto= type's last steps in a real-world scenario.

Figure 13

Prototype behavior t= ilting the platform =

Figure 14 is an internal representation of the actual behavior after finishing the training followin= g a path and choosing the clearest option, depicting the direction that agent 1 will take if it is in the rightmost state, a right inclination. Agent 1 will gradually shift the pole to the left until it reaches the state in row 4. T= he agent 2 detects that is at a state 0, thanks to the sensometer, which means= the platform is facing backward, and it will shift to the front by increasing t= he signal sent to the servo motor by 18⁰, moving from state 0 to state 4= .

Figure 14

Internal Behavior of Agents= =

3D"Imagen

Conclusions

T= his project includes a study of Arduino microcontrollers, some of their peripherals, and the implementation of a RL agent in a completely Arduino-b= ased prototype consisting of two devices that operate by wireless communication. Within the experimental limitations, the automation process implementation = with Artificial Intelligence and RL agents running in Arduino boards, presented = very promising results that, in principle, can extend to more complex intelligent machines in the real world. This will require increasing the number of variables, the resolution of the sensors, the power of the actuators, etc.<= span style=3D'mso-spacerun:yes'> According to the obtained results durin= g the project’s execution, the following facts can be concluded:

· = The restricted memory space = of the Arduino for storing variables is not a barrier to running learning RL algorithms. In particular, Q-learning algorithms can be fully implemented in current Arduino microcontrollers, as demonstrated.

· = In the real world, most envi= ronments are analog; however, as demonstrated in this study, using Q-learning and a = finite Q-matrix it is possible to convert analog changes in discrete signals and u= se them to control a pole balancing situation.

· = Wireless communication adds = value to Arduino projects by allowing them to collect data and/or send instructions = in complex environments with limited space and/or constant movement. Also, ope= ns possible valuable applications in the world of the Internet of things (IoT)= .

· = Agents and RL algorithms are powerful tools because they allow robots to train themselves in many tasks, such as maintaining dynamic balance using only their implemented components such as sensors and actuators.

· = Arduino platforms have acces= s to a variety of peripherals with methodical inclusion, and is a fertile ground to work and develop self-learning macro sensors, robots, and control systems.<= /span>

Referencias

Araúj= o, A., Portugal, D., Couceiro, M. S., & Rocha, R. P. (2015). Integrating Arduino-based educational mobile robots= in ROS. Journal of Intelligent \& Robotic Systems, 77(2), 281–298.

Azade= h, K., De Koster, R., & Roy, D. (2019). = Robotized and automated warehouse systems: Review and recent developments. Transpo= rtation Science, 53(4), 917–945.

Foerster, J., Nardelli, N., Farquhar, G., Afouras, = T., Torr, P. H. S., Kohli, P., & Whiteson, S. (2017). Stabilising experience replay for deep multi-agent reinforcement learning. International Confer= ence on Machine Learning, 1146–1155.

Garcia, J., & Shafie, D. (2020). Teaching a humanoid robot to walk faster through Safe Reinforcement Learning. Engin= eering Applications of Artificial Intelligence, 88, 103360.<= /span>

González, I., & Calderón, A. J. (2019). Integra= tion of open source hardware Arduino platform in automation systems applied to S= mart Grids/Micro-Grids. Sustainable Energy Technologies and Assessments, = 36, 100557.

Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchron= ous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA), 3389–3396.

Hyon, S.-H., Hale, J. G., & Cheng, G. (2007). Full-body compliant human--humanoid interaction: balancing in the presence = of unknown external forces. IEEE Transactions on Robotics, 23(5), 884–898.

Jain, A. K. (2018). Working model of self-driving c= ar using convolutional neural network, Raspberry Pi and Arduino. 2018 Second International Conference on Electronics, Communication and Aerospace Techno= logy (ICECA), 1630–1635.

Jimenez, A.-F., Cardenas, P.-F., Canales, A., Jimen= ez, F., & Portacio, A. (2020). A surve= y on intelligent agents and multi-agents for irrigation scheduling. Computers= and Electronics in Agriculture, 105474.

Korkmaz, H., Ertin, O. B., Kasnako\uglu, C., & others. (2013). Design of a flight stabilizer system for a small fixed wing= unmanned aerial vehicle using system identification. IFAC Proceedings Volumes= , 46(25), 145–149.

Kormushev, P., Calinon, S., & Caldwell, D. G. (2013). Reinforcement learning in robotics: Applications and real-world challenges. Robotics, 2(3), 122–148.

Lengare, P. S., & Rane, M. E. (2015). Human hand tracking using MATLAB to control Arduino based robotic arm. 2015 International Conference on Pervasive Computing (ICPC), 1–4.=

López= -Rodríguez, F. M., & Cuesta, F. (2016). Andruin= o-A1: Low-Cost Educational Mobile Robot Based on Android and Arduino. Journal = of Intelligent and Robotic Systems: Theory and Applications, 81(1), 63–76. https://doi.org/10.1007/s10846-015-0227-x

Mata-= Rivera, M. F., Zagal-Flores, R., & Barría-Huidobro, C. (2019). Telematics and Computing: 8th International Congres= s, WITCOM 2019, Merida, Mexico, November 4--8, 2019, Proceedings (Vol. 1053). Springer Nature.

Miglino, O., Lund, H. H., & Nolfi, S. (1995). Evolving mobile robots in simulated and real environments. Artificial Li= fe, 2(4), 417–434.

Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018). Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. ArXiv Prepr= int ArXiv:1803.11347.

Pan, X., You, Y., Wang, Z., & Lu, C. (2017). Virtual to real reinforcement learning for autonomous driving. ArXiv Preprint ArXiv:1704.03952.

Ram, S. A., Siddarth, N., Manjula, N., Rogan, K., &= amp; Srinivasan, K. (2017). Real-time automation system using Arduino. 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), 1–5.

Salaz= ar, R., Rangel, J. C., Pinzón, C., & Rodríguez, A. (2013). Irrigation system through intelligent agents implemented with arduino technology.

Sharma, A., Ahn, M., Levine, S., Kumar, V., Hausman, K., & Gu, S. (2020). Emergent real-world robotic skills via unsupervised off-policy reinforcement learning. ArXiv Preprint ArXiv:2004.12974.<= o:p>

Sun, M., Luan, T., & Liang, L. (2018). RBF neur= al network compensation-based adaptive control for lift-feedback system of ship fin stabilizers to improve anti-rolling effect. Ocean Engineering, <= i>163, 307–321.

Taha, I. A., & Marhoon, H. M. (2018). Implementation of controlled robot for fire detection and extinguish to clo= sed areas based on Arduino. Telkomnika, 16(2), 654–664.

Wu, D., Liu, S., Zhang, L., Terpenny, J., Gao, R. X= ., Kurfess, T., & Guzzo, J. A. (2017). A fog computing-based framework for process monitoring and prognosis in cyber-manufacturing. Journ= al of Manufacturing Systems, 43, 25–34.