Dickens knows this, and makes some effort to explain himself. In the second chapter David is young, but his jewel-bright recollection continues.
...I think the memory of most of us can go farther back into such times than many of us suppose.
...if it should appear from anything I may set down ... that I was a child of close observation, or that as a man I have a strong memory of my childhood, I undoubtedly lay claim to both these characteristics.
A Family Romance: Here too we start with an aunt: 'the aunt rather than ... my aunt, for anything more intimate would have implied appropriation, or attachment'. Magical stories are invoked: 'Dolly's absence I took for granted, for in the manner of fairy tales I assumed that after the apotheosis it was natural for people to vanish'. And we are treated to detailed recounts of events and conversations, even though the narrator Jane was little more than four or five at the time (and in later chapters she dramatises scenes that took place before her birth).
She tells us, perhaps echoing David Copperfield:
It is not true that children do not understand adult feelings. They understand them all too well, but they are powerless to deal with them.And towards the end of the first chapter is at pains to justify her apparently excellent recall:
These facts were not revealed to me until much later. Many of them I had to supply myself. It seemed to me important to reconstruct the story, even to the point of doing a certain amount of research.