So guys turns out that digging in the dessert is a bit wonky and well ya, breaks the physics? The ground opened up into three diamond shapes from one single hole in the center.

Sandstone seems to not have good enough stability to support itself and doing any excavation in the desert may cause SERVER WIDE physics issues. Please be mindful of this and keep excavation in deserts to a minimum. :)