The quote is from Mario Mencia, author of The Fertile Prison, where it is found on page 36 - though with "their" instead of "our." A footnote references Jose Marti as the source of the "fortress of ideas" image, though without citation.
So, although Mencia is describing Castro's time in prison as a time where study was essential, the words are Mencia's; he certainly would have given credit to Fidel. (Note: Castro did say "prison was our academy of struggle" in a letter to Nico Lopez, 1/1/1955 - see p. 152 of this book. So the sentiment, if not the words, do reflect Castro's attitude towards his prison experience.)