Click here to download this post as Word Document
Document Storage Formats – An Introduction
Document storage databases are all the buzz these days. Interestingly enough they are actually not a very recent invention. Already 20 years ago there were object oriented databases using the very same concepts.
In this four part mini blog series I will take a bird’s eye view at the connection and relationships between object oriented databases, object stores, serialization and document storage. I will present and explain the most common document storage formats and will try to find the reason why object stores are all the buzz today but were not 20 years ago.
Episode 2 – Binary and XML document formats
You could argue that there is no such thing as a binary document storage format. Binary serialization has been around for a long time, not just since developers want to persist objects directly to long term storage (without first having to map between object properties and database columns). When passing objects across process boundaries, which is known as marshaling, developers have to first serialize the objects’ state. In its simplest form the binary document would be a string of bytes representing the values of all variable object properties. For the application to be able to interpret the binary document it had to somehow know about the object’s class definition. The class definition would either be part of the application code, as a declarative class definition, or the application would learn about it from type libraries. Either way, the binary documents were neither humanly readable, nor machine interpretable without explicit knowledge of its source or destination object’s class definition.
Recent implementations for binary document formats are more sophisticated. They contain meta data describing and identifying the structure from which the object’s variable parameters have been serialized. While this increases the resulting document’s size, it makes it more portable and allows its data to be interpreted even after the original object’s class definition has long been forgotten.
Obviously knowing the structure of any document is very useful in many respects. That is why structured document formats contain more or less meta data which self-describes and “communicates” their structure; binary document formats are no exception anymore.
While binary document formats usually contain only very little meta data and therefore are considered the “leaner” of the document formats, XML might be placed at the other end of that scale, being a very “bloated”, or nicer put, a very “verbose” document format.
According to Wikipedia “XML” is defined as follows:
Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards. (Source: https://en.wikipedia.org/wiki/XML#Comments)
XML examples:
<person>
<firstName>John</firstName>
<lastName>Smith</lastName>
<age>25</age>
<address>
<streetAddress>21 2nd Street</streetAddress>
<city>New York</city>
<state>NY</state>
<postalCode>10021</postalCode>
</address>
<phoneNumbers>
<phoneNumber type="home">212 555-1234</phoneNumber>
<phoneNumber type="fax">646 555-4567</phoneNumber>
</phoneNumbers>
</person><person firstName="John" lastName="Smith" age="25">
<address streetAddress="21 2nd Street" city="New York" state="NY" postalCode="10021" />
<phoneNumbers>
<phoneNumber type="home" number="212 555-1234"/>
<phoneNumber type="fax" number="646 555-4567"/>
</phoneNumbers>
</person>The XML document format is probably the most mature of all formats. It is very popular because it is very humanly readable and therefore enables easy debugging of application code working with XML documents. There have been numerous technologies developed to query and manipulate XML documents, XML schemas and other entities related or derived from the XML format. Some database management systems fully support the XML document format as a data type and may even enable transparent indexing of XML elements or attributes. XML is very flexible and versatile, and, by design, very extensible. Nonetheless, when it comes to storing and managing large amounts of object data, XML is not commonly the document format of choice. The CPU overhead for processing and the inflation of data size when using XML documents often make it difficult to justify its use for object storage. Applications often cache object documents in memory, perform searches or index operations in memory. The greater the size overhead from your document format, the fewer object documents fit into a finite amount of memory. Therefore application developers seek more compact document formats and avoid XML object storage in such use cases. Still, to store configuration data for a relatively small count of entities, the XML document format is very suitable – and popular.
Preview of next week’s Episode 3: In my next blog episode I will look closer at the JSON and BSON document formats. Currently those are probably the most popular document formats. MongoDB, for example, uses both formats: BSON for internal storage and JSON for communicating with your application.
Resources and references: If you are interested in more information please visit the following links:
https://en.wikipedia.org/wiki/XML_Namespace
https://en.wikipedia.org/wiki/XML_Base
https://en.wikipedia.org/wiki/XPath
https://en.wikipedia.org/wiki/XSLT
https://en.wikipedia.org/wiki/XQuery
https://www.w3schools.com/xml/xml_whatis.asp
https://msdn.microsoft.com/en-us/library/72hyey7b(v=vs.71).aspx
