We propose a privacy-preserving image classification method based on perceptual encryption that does not require centralized key management. In the proposed method, each client independently generates an encryption key to protect visual information in both training and query images. The use of independent keys allows multiple clients to use a shared model without exchanging keys and to easily update their keys whenever needed. In addition, even if a key is compromised, the impact does not propagate to other clients. The use of perceptual encryption allows us to directly apply encrypted data for training and query images in the encrypted domain, but conventional approaches with perceptual encryption are known to degrade the accuracy of image classification when independent keys are used in each client due to significant visual distortion caused by encryption. Accordingly, we demonstrate that a novel method that focuses on the compatibility between block-wise image encryption and the embedding structure of vision transformer (ViT) is effective in improving the issue. We carried out experiments to demonstrate the effectiveness of the method in terms of accuracy and robustness on CIFAR-10 and Tiny ImageNet. Compared to conventional methods, when using independent keys, the accuracy was improved by 82% for CIFAR-10 and 83% for Tiny ImageNet. In addition, resistance to various attacks including brute-force attacks and jigsaw puzzle attacks was demonstrated under the assumption of ciphertext-only attacks. These results suggest the practicality and effectiveness of the method for secure image classification in real-world multi-client environments.