My first attempt at a definition was this: a story containing one or more internal narratives. The internal narrative(s) may involve multiple narrators, as in The Decameron, or just one, as in Heart of Darkness.
Thinking about it, though, I realised that there are plenty of stories involving internal narratives where I wouldn’t be happy to call the context in which those narratives appear a frame story. For example: in Oedipus Rex a messenger narrates Oedipus’s self-blinding. It would be strange to refer to the rest of the play as a frame story for that narrative. Is it just because that narrative comprises such a small portion of the play? On the other hand, in the Odyssey Odysseus’s account to the Phaeacians of his travels takes up a substantial portion of the poem (books 7-12), but I still wouldn’t want to call the rest of the Odyssey a frame story for that narrative. If there is a minimum proportion that must be taken up by the internal narrative(s) for the remainder to be considered a frame story, then what is that proportion?
Or is it also to do the with the relationship of the internal narrative to the outer narrative – i.e. it must have a certain independence from it? The Wife of Bath’s tale doesn’t have much to do with making a pilgrimage to Canterbury, for example, but the account of Oedipus’s blinding is intimately related to what happens in the rest of the play - which must surely be an additional factor in helping to disqualify their relationship as one of story to frame?
Does the identity of the internal narrator have a bearing on all this? In The Canterbury Tales the tales are told by characters, but in, say, Louis Sachar’s Holes, the lengthy backstories of the various characters are (iirc) all told by the third-person narrator. Does that continuity of voice make the rest of the book less framelike?
Finally, does it matter whether we return to the frame story at the end of the internal narrative (as in Conrad), or neglect to, as in The Turn of the Screw? If you are still inclined to say that James’s story has a frame, does it matter that the frame is “missing” a piece? If it doesn’t, what about an example such as Ovid’s Metamorphoses, which is just a chain of stories linked together, sometimes segueing in apparently “framey” ways, but in which there is little nesting, and few or no pushes gets popped (to use the language of computer stacks). You might then think of the Metamorphoses as one big frame story, and/or one big internal narrative at the same time – a kind of narrative Möbius strip where the inside and outside are the same.