The notion that federated learning ensures privacy simply by keeping data local is widely acknowledged to be flawed. Cryptographic techniques such as Multi-Party Computation (MPC) and Fully Homomorphic Encryption (FHE) address this issue by concealing the model during the training procedure, but their extreme computational and communication overhead makes them impractical for real-world deployment.

However, we argue that such strong guarantees are unnecessary. Even with full-model encryption, black-box attacks remain possible during the prediction phase, since model outputs are eventually revealed to the querier. This suggests that instead of enforcing perfect privacy during training, it is sufficient to ensure that the leakage during training is no higher than the leakage during prediction.

To achieve this, we generalize POSEIDON (NDSS 2021), a state-of-the-art FHE-based federated learning approach, by selectively encrypting only the components of the model necessary to match the privacy level of the prediction phase. Our method identifies the parts of the model that contribute most to information leakage and prioritizes their encryption, significantly reducing computational and communication overhead.

Our experiments on dense neural networks show that encrypting only the last layer is often sufficient to hinder white-box attacks, improving efficiency by a linear factor in the number of layers. For deeper models, multiple layers may require encryption, but our approach still achieves a substantial speedup compared to full-model encryption.