There is a natural relationship between the arrows and the movement that the user expects when tapping on an object that contains it. Usually, the up and down arrows are used to scroll or display content, especially in Android, while the side arrows are often used to indicate forward and backward movement between screens. In iOS, it is common to find them in menus of options:
iOS Settings uses arrows to access submenus. The screens appear from the right. Back button shows an opposite direction arrow (source: Screenshots from iOS Settings)
Note that not all languages follow a left-to-right reading direction, usually written as LTR. Those languages whose direction of reading is RTL can interpret the arrows as contrary to the expected one.