He hadn't actually arrived late, but when they descended the cliff, he didn't follow them down.
Because he discovered that others had gone down the cliff after them, he stayed hidden at the top, waiting to see how the situation would unfold.
Unexpectedly, once they came back up, they parted ways. For a period of time, he had lost track of Chu Yueli.
Chapter 672