cky-mongo-decay

cky-mongo-decay is a tool that helps you to split your mongodb into smaller mongodb to serve the needs of fragmented storage. Use environment DEBUG=DecayMongo for show log debug

All bug or issue please send to email: [e-mail]

Features!

• split your mongodb into multi mongodb with your special condition
• use mongoose package for upsert document into your pieces mongodb

Installation

$ npm install cky-mongo-decay --save

function

• contructor (options)

  • fnDecay *** (require)***: The function used to filter the document into mongodb you want
    • example:

      fnDecay: (obj) => {
          return moment(obj.publishDate).utcOffset(420).format('YYYYMM');
      }

  • schema *** (require)***: Use for upsert your mongodb
  • modelName *** (require)***: Model name, use for upsert your mongodb
  • piecesOfDecay *** (require)***: Object setup your pieces mongodb with :
    • key: result from fnDecay, which you want your document was filtered
    • value: Connection string of your piece mongodb
    • example:

      piecesOfDecay: {
          '201907': 'mongodb://user:pwd@host:port/db1',
          '201908': 'mongodb://user:pwd@host:port/db2',
          '201909': 'mongodb://user:pwd@host:port/db3',
          '201910': 'mongodb://user:pwd@host:port/db1'
      }
      

  • limitDecay: ******: limit async run in parallel (lib use package async), default is 5
  • stopDecayWhenError ******: if you want skip error when upsert into your pieces mongodb then set it is false, default is true (Stop and return error callback when error happen)

• init (callback)

  • Run first for init this lib, it will process connect all connection of your peices mongodb, callback when done with format (err, result)
  • callback: error when one of host connect fail
  • after init success, you can use this lib

• decay (findObj, arrObjDecay, callback)

  • findObj : use for query upsert,
    • example:

      {
          slug: 1
      }
      

    • this example above mean: query key slug with value of document you want upsert into your peice mongodb, it will become:

      {
          slug: 'this-is-slug-for-find-query-of-mongodb'
      }
      

  • arrObjDecay : list multi or one document you want insert to your peice mongodb, if you pass Array then result in callback is Array[String] else is String
    • example:

      {
          "slug" : "this-is-slug-for-find-query-of-mongodb",
          "updatedAt" : new Date("2019-11-21T16:55:00.794+07:00"),
          "createdAt" : new Date("2019-11-21T12:19:02.454+07:00"),
          "title" : "Hello world!",
          "publishDate" : new Date("2019-11-27T14:00:00.000+07:00")
      }
      

  • callback : format (err, result)
    • err: error when process fail and flag stopDecayWhenError is true
    • result: is Array[Object] if arrObjDecay is Array else Object. Element in Array is object have field decayId is new ObjectId mongodb, in case you use stopDecayWhenError = false and upsert error then decayId is null.
      • example:

        [
            {
                "find": {
                    "slug": "slug-1"
                },
                "decayId": "5dd83b2bd82b317c61758d61"
            },
            {
                "find": {
                    "slug": "slug-2"
                },
                "decayId": null
            },
        ]
        // or
        {
            "find": {
                "slug": "slug-2"
            },
            "decayId": "5dd83b2bd82b317c61758d61"
        }
        // or
        {
            "find": {
                "slug": "slug-3"
            },
            "decayId": null
        }

• close (callback)

  • Close all connection of your peices mongodb you was setup when init
  • callback is function (err), err != null when close error

Usage Example

const moment = require('moment');
const DecayMongo = require('cky-mongo-decay');
 
const decay = new DecayMongo({
    fnDecay: (obj) => {
        return moment(obj.publishDate).utcOffset(420).format('YYYYMM');
    },
    schema: require('./FeedSchema'),
    modelName: 'Feed',
    limitDecay: 1,
    stopDecayWhenError: false,
    piecesOfDecay: {
        '201907': 'mongodb://user:pwd@host:port/db1',
        '201908': 'mongodb://user:pwd@host:port/db2',
        '201909': 'mongodb://user:pwd@host:port/db3',
        '201910': 'mongodb://user:pwd@host:port/db1'
    }
});
 
let obj4Decay = [{
    "slug" : "slug-1",
    "title" : "Hello world!",
    "publishDate" : new Date("2019-07-27T14:00:00.000+07:00")
}, {
    "slug" : "slug-2",
    "title" : "Hello world!",
    "publishDate" : new Date("2019-08-27T14:00:00.000+07:00")
}, {
    "slug" : "slug-3",
    "title" : "Hello world!",
    "publishDate" : new Date("2019-09-27T14:00:00.000+07:00")
}, {
    "slug" : "slug-4",
    "title" : "Hello world!",
    "publishDate" : new Date("2019-10-27T14:00:00.000+07:00")
}, {
    "slug" : "slug-5",
    "title" : "Hello world!",
    "publishDate" : new Date("2019-11-27T14:00:00.000+07:00")
}]
 
decay.init((err, result) => {
    if (err) {
        return console.log('init fail, err=', err, 'result=', result);
    }
 
    decay.decay({
        slug: 1
    },
    obj4Decay,
    (err, result) => {
        console.log('done err=', err);
        console.log('done result=', result);
 
        decay.close(() => {
            process.exit(0);
        })
    })
})

result will like this:

[
  {
    "find": {
      "slug": "slug-1"
    },
    "decayId": "5dd83b2bd82b317c61758d61"
  },
  {
    "find": {
      "slug": "slug-2"
    },
    "decayId": "5dd83b2bd82b317c61758df8"
  },
  {
    "find": {
      "slug": "slug-3"
    },
    "decayId": "5dd83b2cd82b317c61758ea5"
  },
  {
    "find": {
      "slug": "slug-4"
    },
    "decayId": "5dd83b2cd82b317c61758ea5"
  },
  {
    "find": {
      "slug": "slug-5"
    },
    "decayId": null
  }
]

License

ISC

- Chickyky -