I want to rename S3 objects atomically.

Asked 2 years ago, Updated 2 years ago, 163 views

It's gzip compressed in a specific bucket, but the extension is saved as .json.
If things go on like this, the batch process will not recognize it as a gz file, so
I want to rename .json.gz

However, it is already in service and there is always a write from the API in this bucket
json.gz presence check → renaming is not atomic, so

if update processing from API is interrupted during rename processing after existence check I will overwrite the latest data with old data, so I would like to prevent that from happening

If it were a file system, I would do something like lock.
Are there any locks on S3 objects?

Or, it seems that the renaming user can create but not update
I thought it would be possible if I had IAM permission.
S3:PutObject permission appears to be the only one for object writing

Could you give me some advice on what to do in this case?

amazon-s3

2022-09-30 18:19

1 Answers

In the first place, S3 does not have a rename API, so it is a copyObject->deleteObject operation.
In the case of copyObject, of course, if there is an object with the same name, it will fail, so I think you should copy it as another name.

If you can't stop the service, it can't be helped, but if you can, I think it's best to stop it.


2022-09-30 18:19

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.