I don't know many people who "Safely Eject" their removable storage media.
Even I just pull them out when I think I know the risks.
Ideally you should perform the Safe ejection process every time you remove a USB thumb drive, an SD card, external hard disk or any other writable type of storage media.
This is especially so when you are using those things to store backups or move otherwise important data.
Your computer may look like it is finished with your device but in reality it could still be tidying things up in the background or in some other way still have its hooks into the device.
The Safe Eject helps you and your computer finish up and make sure all is finished before you interrupt something important.
Removing removable storage before it is safe to do so is like taking the hose out of the bucket before turning it off...
...what you wanted in the bucket is now "somewhere else".
If this was your files, then you've just successfully corrupted them and made them unusable.
The problem is that sometimes the Safe Eject never seems to let us know that it is OK to remove the device.
You can close programs that may be using the files, you can close the file explorer...maybe your antivirus product is still scanning it?
But once you've done all the reasonable checks, and if you really, really must do a safe ejection, then shutting down your PC is the only option. But that's a bit drastic.
We end up being forced to remove the device regardless of how "safe" it is.
At least now you know the risks and can check that your important files are OK. P.S. You need to know that thumb drives and the like ARE fallible. Don't store just the one copy of your life's work on any removable media. Back it up properly and create versions as you go.