If we're going to have a good chance at taking a castle we need a lot more men then we currently have. For this, we'll need to capture more people. Even a weak castle won't be easily taken with numbers below 50 men; those walls provide a huge advantage to them.