What is meant by up to X GHz?

For example, an AMD Ryzen 3750H has a base clock frequency of 2.3 GHz, but it says that it goes up to 4 GHz.

Do you have to overclock the CPU?

Respectively. If so, are AMD Ryzen processors worth it for laptops at all?

Most processors can limit themselves to the max. Frequency up

No there's a so-called base block and a turboclock.

The turboclock is used when the CPU has a lot to do, otherwise the base block is used.

In your case 2.3 GHz is the base block. The 4 GHz can - but does not have to - be reached. This requires the performance and the processor must not be too hot.