ARRGHH! I wrote up a reply and it disappeared! Let's see if I can reconstruct it. :(
The song plays a vital role in the film. Esther and John are attracted to each other immediately, but after the initial attraction, there has to be something more to make this the beginnings of real love. When John sees Esther on the staircase bathed in the copper glow of the dimmed lights, he is overwhelmed by her loveliness and feels a need to express those emotions, and he thinks of the lyrics to the song, and tries to recall them. He has an appreciation of her beauty, and furthermore, an appreciation for language used to describe such beauty. When Esther sings, she is acknowledging that they have this song and this sensibility in common and that she understands what he is trying to say. Meet Me In St. Louis is not a fluffy film. Mere "boy meets girl and falls in love" won't cut it. There has to be a deeper and genuine connection between Esther and John and the song creates that connection.
That said, an actual song at that very delicate moment does seem a bit intrusive. I probably would have had John grasping for a line of poetry he had liked, and then Esther completing the poem. But either song or poem, something needs to be there, and early on in the film, so we *really* root for Esther and John as a couple not because the movie says they are a couple, but because they truly belong together.
As for the business with the lights, that is no trick. *Both* Esther and John know it's a lame excuse, but it's a perfect one for the situation. It's 1903 St. Louis and no proper young lady would entertain a young man in the house late at night when the parents are both out and everyone else has gone to bed. Esther's request provides an excellent reason, no matter how flimsy, for John to linger. In fact, he would be a cad to refuse. ;) And if he did see the lights go back on after he left, if anything, he would have been amused.
Besides, the lights have to be turned down low so we can have exquisite pictures like this, with the moonlight suddenly coming through the etched, colored staircase window:

Or this...

Esther's on the trolley but she's not jolly because John is missing.


Mystery Girl alert! There she is on the right. But the real mystery is, who came up with the color combination on that girl in the middle? Puce with PINK buttons and bows? Ugh! In fact, there is a lot of puce in this movie. On the other hand, Esther's simple black velvet jacket and lack of hat make for the perfect contrast to all the pastel ruffles and furbelows on the other girls, and the audience's eye is drawn straight to her.

Prominent moment for our Mystery Girl.

He made it!

It's Hallowe'en so Tootie is up to mischief.

My bulldog Charlie insisted I post the next two pictures.

