Others are clearly as ignorant as yourself when suggesting that any patch of land has a theoretical maximum production capacity. There is no theoretical maximum production capacity, *especially* not in agriculture. You can always develop more productive breeds, invent better fertilizers, discover better crop management, etc.
Ooook.... First of all, you will probably remember from secondary school biology that for any population of living beings there is a carrying capacity above which the species begins to even out in population. This applies to
all living things, and even if you didn't like botany (don't blame you, I didn't like it either), you'd still know that plants, even domesticated (er, are they actually called domesticated?) ones, also must fit within the constraints of a particular region's carrying capacity. The scope can change, whether it be a patch of arable land in a given location, or whether we are talking about Earth's biosphere's maximum capacity.
There is a maximum. How is this achieved? Well for one thing, growing too many crops together will place a strain on nutrients. There's one of your limiting factors. Can you come up with enough nutrients in the soil to support all those crops? The mere decomposition of crops does not yield anywhere near a 100% efficient recycling system. Sure, you can improve this ratio, and even get near saturation on your field.
Then comes your second limiting factor: space. You are aware that all matter occupies space (yeah, let's stick to very basic science please) and your crops are no different. Even if you manage to grow them on atop the other, their roots will still need space to impact into the soil and receive nutrients, and, just so you know, plants do respire as well. Cramping together would mean some plants might not have the proper gas exchange, and in terms of the light cycle, some plants might never get any sunlight, thus killing them because they don't have the means to conduct photosynthesis. Since the vast majority of food crop are dedicated photoautotrophs, that means they cannot synthesize their energy source without light.
Third, we have to consider our maximum scope. Even if we can make the entire Earth's surface arable, and somehow we learn how to create floating farms in the atmosphere, we cannot grow anything in the molten layers beneath the crust, and we cannot grow in the vacuum of space. If you want to talk about extraterrestrial agriculture, then unfortunately you are outside your desired realism, because BM takes place on the surface of whatever planet or world upon which it takes place. So we are limited ultimately by how much space we have in our potential "system".
I'm sure there are a number of further limiting factors that would make a carrying capacity have an actual practical limit, such that, short of figuring out how to get around the laws of our universe, we cannot further increase crop yield. Quick examples would be disease, pests, workforce, climate.