Far from being a straightforward narrative, events are all over the place from a temporal point of view, and this is a novel in which little happens in terms of external actions: the psychology of the two principal individuals is all-important. There is the teacher Pierre, who has two children by his wife Annie, who works in a bank; and then there's Laure, who's an unmarried librarian having an affair with Pierre.
Laure is faithful to Pierre, who really calls all the shots: he calls Laure and calls on her when he can, which is mainly in the afternoons when Annie is working, and although the pair can occasionally sneak away for a brief time they remain at Laure's place mostly as they might be seen together by friends and acquaintances in the unnamed (and probably small) town.
The expression 'La vie fantôme' (or 'The Ghost Life') in fact has several meanings. To Laure it means being tied to the schedule (and often the whim) of Pierre, of being available at her home or able to change her work hours with a colleague often at the drop of a hat in order to be ready for a few hours of love (or lust, or whatever) with Pierre: her life certainly isn't empty without him, but it frequently feels severely reduced, almost a non-life.
For Pierre it's obviously different as he's leading a kind of double life in that he has to fulfil his role both as a husband and a father and at the same time juggle his hours to fit Laure in. He sometimes feels that he has a complete life, but then there are obligations towards everyone. He's leading a kind of ghost life too, although he's evidently also having his cake and eating it. And sometimes he can be a really inconsiderate bastard, not considering Laure's feelings: just how could he be so unthinking as to give Laure a diary as a present, when it's obviously come from Annie's bank? And how can he give her flowers that he's picked from his garden, when this garden belongs to the other life, it's an intrusion on the life they have together?
Finally, the ghost life is seen as what many married couples have, going through the motions of happiness because they're too lazy (or something) to break free.
Towards the end there's a little more action when Pierre goes to Normandy to a vacant house of a friend of a work colleague of Annie's and Annie can't join him the first week so Pierre calls Laure to join him and they enjoy a rare days together. But then Annie arrives unexpectedly when the couple are lunching at a restaurant, Pierre calls back at the house to find the wallet he's forgotten but also finds Annie (who doesn't have a key) has just arrived. So he hastily concocts a story about losing the key while Laure tidies up the house, collects her clothes from the wardrobe, sees to it the crockery and glass situation look unsuspicious, and then drives back home in an understandable numb daze. And then several days later Annie (who doesn't normally notice these things) says what a coincidence it was that there was a car immediately outside the house bearing a number plate from their département. So has she just suspected, or has she known all along?
This book is a large number of things, but it struck me as the biggest argument against adultery that I've come across.