How to decrypt encrypted files without key? Is there any specific process that I need to follow?
Is it possible, in theory, to work through each byte and try to figure out what it originally was without a key, or brute-force? Is there any kind of strategy or pattern that can enable one to reverse the data manually?
The answer to how to decrypt encrypted files without key is you can't. The only proven safe "encryption" is a one-time pad but that's very impractical... I'm going to save you the long, technical story.
You have probably heard of some real world encryption algorithms: RSA, AES, RC4, etc. The thing is that we do not know whether any of these are secure, we only know that they've been around for a long time and not yet broken despite many, many attempts. Well, cross out RC4 in the list of safe ones: cryptanalysts' relationship status with RC4 is "it's complicated".
Let's take AES. It translates bits of data into other data. You put "CAT" in, give it the key "4ZD" and "YYR" comes out. You give it "YYR", provide the key "4ZD" and "CAT" comes out. Of course there is more to it than that, but that's the idea.
If AES would be completely secure, you would need to try all possible keys (in this case AAA through ZZZ and 000 through 999) in order to find that "4ZD" is the correct encryption key. For AES, there are attacks known by which you can find a small bit, for example you can tell that one of the key characters is a digit but you don't know which digit or in which position. Still many possibilities to try.
But this is the best we've been able to do in the 16 years that AES exists. It seems unlikely that it will ever be fully broken. There are security features in AES against things that went wrong in older encryption algorithms:
It is, so far as we know, secure against known plaintext attacks. This means that given "CAT" and "YYR", you cannot learn that the key is "4ZD". Why is this useful? Well consider websites: many web pages contain English words. If you intercept https data and run it against an English dictionary, one of the keys that you get would be the correct one for the rest of the webpage. But as I said, this is not possible with AES.
AES is, so far as we know, also secure against chosen plaintext attacks. If you are allowed to choose what is going to be encrypted and are then able to see the encryption, you can still not determine the encryption key, thereby still being unable to decrypt the rest of the data.
When using similar but not identical passwords, such as P@ssw0rd! and P@ssw0rd1, the encryption result is completely and unpredictably different. Or given similar but not identical texts, it would also be completely different.
So things like guessing the password character for character are not possible. You'd need to get the whole password correct at once. The average number of attempts this takes is (2^password_security_in_bits)/2. Take it from me that with AES-128, this is impossibly long. (Of course 4ZD is not a realistic password.)
There are many more of these known attacks, which our modern encryption algorithms try to protect against of course. The only way to break mainstream encryption is to circumvent it (as far as we know). It might be that the police in your fiction series previously installed a keylogger to record the password from the suspect. It might be that they have a camera recording where the suspect enters his password. Or he might have used a weak password that was crackable in seconds.