The new world of cloud storage - do you really need ALL your old data?!

Traditionally, when moving from one fundraising database to a new one, charities tend to migrate all their old data across - or very nearly all. Sometimes, small sub-sets of old or truly unneeded data may be left behind, but in the main, every single donation, communication, appeal record etc and every entity and every field is transferred as is to the new database.

Why? Well, understandably, fundraisers and marketing staff are worried about losing granularity of historic data, of not being able to do true and useful RFV calculations or similar analysis on summary fields, of not being able to look back at why someone responded to appeal X or came to event Y. And the thought of either not keeping their data in the most granular state possible, or even not transferring it at all, has been a complete anathema!

But recently, this is something which is being challenged. This is partly due to cloud CRM systems charging for extra storage, partly due to some performance concerns for complex processes on very large systems and partly to mitigate some of the risks and complications of data migration. And partly it is just good business practise to challenge such assumptions.

But Why?!
But just as pertinent a question for me is: Why - Why do you need ALL your old data? Really - why?! I mean, are you really going to look back at someone's communication history from ten years ago and want to see that a donor was sent a second letter because a particular acknowledgement letter didn't reach them first time? Do you really need to know that someone was invited to an event twelve years ago but didn't respond? I'm not sure if you do…

And although I completely recognise the need to keep a granular donation history for at least a set number of recent years (e.g. at least four years for gift aid claims), at what point are your fundraisers not going to use such data for segmentation and marketing analysis? Five years? Six maybe? In which case, do you really have to be able to see each individual donation for older periods as a separate financial transaction on your new system? Or actually, could a summary be acceptable? e.g. a summary by year/campaign/gift type/whatever is appropriate for you.

Of course, there are some data items you would definitely need to keep, such as first donation information, maybe upgrade details etc, but do you have to have more than that for donations over a certain age?

And if you do, if you have to or if your organisation is too paranoid not to keep such data, then maybe you can store it elsewhere for when (if!) it is needed. For example, a separate data warehouse, maybe a simple SQL Server database, even a separate marketing software system.

And what about any old data which you still have on your system but which almost no-one in your organisation trusts and so doesn't use anyway? Does that really need to be transferred? Really?! And data accuracy in general - just how good is it over a certain period anyway?

Or, Should we be asking, Why Not?
Now. One could of course ask why we shouldn't migrate all data and that's a fair question. Storage does cost but is it so much more? And contemporary systems are so much more efficient now than older systems, why shouldn't we have millions of records in our new database?

Well, maybe because more data does add extra data management, does add complexity when doing segmentation and reports/analysis, can slow down a system when running such processes, can make screens slower to load when they have lots of historic information, can make data migrations more complex etc etc.

I think it is an interesting challenge - one I encourage you to consider!