Couchbase What is ideal approach to archive and delete documents
Problem Description:
What will be suggested approach for archiving couchbase documents based on certain criteria to secondary DB considering secondary cluster is also couchbase.
After moving those document to secondary cluster it should be deleted from primary database that is the requirement.
What will efficient approach to move millions of documents to other couchbase cluster (secondary) and then delete from primary database.
Solution – 1
If you want the archiving to be continuous, you could configure Cross Data Center Replication (XDCR) to the secondary cluster, with a filter that ignores document expiry and deletion.
Specifically, enable:
- Do not replicate document expirations
- Remove TTL from replicated items
- Do not replicate DELETE operations
Then set a max-TTL on the bucket in the source cluster so the documents there expire automatically. Alternatively, delete documents from the source cluster on your own schedule.
CAVEAT: You’ll want to keep a close eye on the XDCR status to make sure documents are replicated before they are deleted/expired.