Click here to download this post as Word Document
Document Storage Formats – An Introduction
Document storage databases are all the buzz these days. Interestingly enough they are actually not a very recent invention. Already 20 years ago there were object oriented databases using the very same concepts.
In this four part mini blog series I will take a bird’s eye view at the connection and relationships between object oriented databases, object stores, serialization and document storage. I will present and explain the most common document storage formats and will try to find the reason why object stores are all the buzz today but were not 20 years ago.
Episode 1 – Introduction and definitions
Document Storage: A document in context of software development is considered, plain and simple, a computer data-file or data-set. For different purposes the data file or set might contain different content. Often, when the content of the document requires a predictable structure, meta data is being introduced to define the document data’s formatting. In such case you consider the document’s data to be structured, otherwise unstructured. An example for unstructured data is a word document; an example for structured data is an XML document.
An application has different options as to where to store its document’s data. Most commonly applications employ a combination of disk storage and memory caching. Obviously the choice for storage will impact performance and scalability of your application. In more recent days there are also options to store the data in the cloud or on solid state disk (SSD).
Object Storage: Within the application all object instances live in random access memory (unless they are memory mapped to e.g. your swap file, which for now, we assume they are not). If the power goes down your random access memory will lose all its data and therefore all of the application’s object instance data (amongst other data). If you want to hold on to that data you have to persist it; that is where Object Storage comes into play – as an option to persist your application’s object instance data to.
Most software and web development frameworks (for example .Net) provide ready-to-use, boilerplate source code or complete implementations of object interfaces for your custom classes which enable you to easily persist their instance data to a document of a particular structure. Such code automatically takes care of finding your instance data and converting it to the appropriate format (e.g. for date/time data types). Some of them even take enumerations, complex data types and even object hierarchies and arrays into consideration. Common are implementations for Binary format and maybe XML. But online you can find boilerplate source code to persist your objects to pretty much any format you have ever heard of.
Some challenges for Object Storage are dealing with object versioning and object inheritance. Also handling Object references rather than instances can be tricky. Don’t assume that those “advanced features” are naturally implemented in all development frameworks. Always make sure to verify before using; and benchmark after coding. Default implementations are often not the best performing ones.
Serialization: According to Wikipedia “serialization is the process of converting a data structure or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and “resurrected” later in the same or another computer environment.”
(Source: https://en.wikipedia.org/wiki/Serialization)
Most commonly serialization is referred to as what you have to do when you want to persist an object instance in your application onto a storage medium other than memory. The basis of the motivation for serialization is that in opposite to random access memory, all other storage types (like disk, cloud or SSD) do NOT provide a fast mechanism to access any byte, anywhere at any time. Performance suffers heavily when trying to read e.g. from a harddrive using random access. In order to make up for some of the natural performance inferiority of e.g. harddrive storage, it is better to “stream” the data to and from the storage device in a sequential access fashion; therefore, at the end of a serialization you usually receive a structured document, which is suitable for “streaming” it to its storage destination, which is document storage.
It is obvious that serialization comes at a price. Depending on which document storage format you choose you observe different impact on CPU processing utilization and transmission bandwidth. Serialization tends to “bloat” the amount of data you have to transmit and store.
Sometimes memory mapped disk storage can be an alternative to serialization but, I believe, this is not widely used. Maybe as SSDs become cheaper and faster there may be a day when memory mapping becomes a viable alternative to serialization (I have pitched this idea to FusionIO but they didn’t seem to be impressed).
Common Document Storage Formats: As mentioned before there are many different document formats for structured document storage. Most famously probably XML and Binary. With MongoDB becoming a “household document store” JSON and BSON are also becoming more widely known (yes, I know, the Java programmers out there will disagree with me that it took MongoDB to make JSON famous); and a more exotic one is Protocol Buffers, a very compressed, binary, structured document storage format introduced and used by Google.
Preview of next week’s Episode 2: In my next blog post I will look closer at the XML and Binary document formats. Also I will start to look at different aspects of storage formats in general which have an impact on performance and scalability and provide reasons as to why one format might be more suitable for a certain use case than another.
Resources and references: If you are interested in more information please visit the following links:
https://github.com/google/protobuf
https://msdn.microsoft.com/en-us/library/72hyey7b(v=vs.71).aspx
https://en.wikipedia.org/wiki/Serialization
http://www.cs.cornell.edu/info/people/chichao/ccc-ch5.pdf
http://www.boost.org/doc/libs/1_35_0/libs/serialization/doc/special.html
https://dzone.com/articles/object-serialization-evil
