Couldn't you just remove
the lattice from the top?

Not really. You'd have to see the fence to see what I mean, it would look ugly. And we'd lose the privacy we were trying to get.