Caves along seacoasts generally have been gouged out by water erosion.
Most of these caves are made of limestone, a relatively soft rock. Waves crashing against rock face for hundreds, thousands, perhaps millions of years will wear away part of the rock.
Depending on the force of the water, the composition of the rock, and the frequency of the waves, a cave may be created.
Inland caves are also formed by water erosion. Carbon dioxide, an element found in our atmosphere, mixes with rainwater to form a mild acid.
The acidified rainwater makes its way through cracks in the limestone and, over time, carves caves from the rock.