Ensure you have the following setting in the Admin > System > Settings. This ensures the customer can still checkout for out of stock products.
Now, go to Admin > Catalog > Products. Edit the products for which you want to restrict checkout and enable “Do not allow checkout when qty is 0“. Click Save.
Now, try testing to place an order for those products. You should not be able to reach checkout page, it should always redirect to the cart page.