A lot of Dockerfiles start with installing dependencies via the base image’s package manager, without specifying exact versions (which isn’t always possible, as most distros don’t keep all history of all packages in their repos). So all your dependencies may have different versions, when you build again.

I’m not sure if supports encryption though, which is probably where a dedicated server would be useful.

Well, ideally you encrypt your data before transferring, so the provider never sees your data. I’m using a storagebox to backup btrfs incremental snapshots (using btrbk) and just AES encrypt them locally before sending them over, so I don’t care if the storagebox itself is encrypted.