Jun
28
2017
--

The next generational shift in enterprise infrastructure has arrived

 Cloud computing is driving growth at 3 of the 5 most valuable companies in the world. AI will impact jobs only as quickly as AI-powered business software evolves. These are just two of the ramifications of disruptions in enterprise technology permeating mainstream media. Yet the inner workings of the tightly knit enterprise software industry are rarely publicized. Read More

Jun
15
2017
--

VR’s killer app: business services

 Enterprise adoption is trumping entertainment uses and will spring VR and AR into the mainstream. Read More

Apr
07
2017
--

Tracking the explosive growth of open-source software

 Many hot new enterprise technologies are centered around free, “open-source” technology. But how can corporate customers, and investors, evaluate all these new open-source offerings? These questions are especially tough to answer because most open-source companies are still private. That’s why we created a detailed index to track popular open-source software projects. Read More

Dec
29
2016
--

How viral open-source startups can build themselves into enterprise-IT powerhouses

Striped Halftone Pattern Hordes of new enterprise-IT upstarts have popped up in Silicon Valley, with some drawing lofty valuations from investors. They’re driven by new, more-advanced technologies in areas such as databases, software development, networking and cloud computing. And many are taking aim at incumbents like Dell, EMC, Oracle and IBM. But will these new companies ever be as valuable as those big names? Read More

Dec
14
2016
--

Row Store and Column Store Databases

Row Store and Column Store

Row Store and Column StoreIn this blog post, we’ll discuss the differences between row store and column store databases.

Clients often ask us if they should or could be using columnar databases. For some applications, a columnar database is a great choice; for others, you should stick with the tried and true row-based option.

At a basic level, row stores are great for transaction processing. Column stores are great for highly analytical query models. Row stores have the ability to write data very quickly, whereas a column store is awesome at aggregating large volumes of data for a subset of columns.

One of the benefits of a columnar database is its crazy fast query speeds. In some cases, queries that took minutes or hours are completed in seconds. This makes columnar databases a good choice in a query-heavy environment. But you must make sure that the queries you run are really suited to a columnar database.

Data Storage

Let’s think about a basic database, like a stockbroker’s transaction records. In a row store, each client would have a record with their basic information – name, address, phone number, etc. – in a single table. It’s likely that each record would have a unique identifier. In our case, it would probably be an

account_number

.

There is another table that stored stock transactions. Again, each transaction is uniquely identified by something like a

transaction_id

. Each transaction is associated to one

account_number

, but each

account_number

 is associated with multiple transactions. This provides us with a one-to-many relationship, and is a classic example of a transactional database.

We store all these tables on a disk and, when we run a query, the system might access lots of data before it determines what information is relevant to the specific query. If we want to know the

account_number

,

first_name

,

last_name

,

stock

, and

purchase_price

 for a given time period, the system needs to access all of the information for the two tables, including fields that may not be relevant to the query. It then performs a join to relate the two tables’ data, and then it can return the information. This can be inefficient at scale, and this is just one example of a query that would probably run faster on a columnar database.

With a columnar database, each field from each table is stored in its own file or set of files. In our example database, all

account_number

 data is stored in one file, all

transaction_id

 data is stored in another file, and so on. This provides some efficiencies when running queries against wide tables, since it is unlikely that a query needs to return all of the fields in a single table. In the query example above, we’d only need to access the files that contained data from the requested fields. You can ignore all other fields that exist in the table. This ability to minimize i/o is one of the key reasons columnar databases can perform much faster.

Normalization Versus Denormalization

Additionally, many columnar databases prefer a denormalized data structure. In the example above, we have two separate tables: one for account information and one for transaction information. In many columnar databases, a single table could represent this information. With this denormalized design, when a query like the one presented is run, no joins would need to be processed in the columnar database, so the query will likely run much faster.

The reason for normalizing data is that it allows data to be written to the database in a highly efficient manner. In our row store example, we need to record just the relevant transaction details whenever an existing customer makes a transaction. The account information does not need to be written along with the transaction data. Instead, we reference the

account_number

 to gain access to all of the fields in the accounts table.

The place where a columnar database really shines is when we want to run a query that would, for example, determine the average price for a specific stock over a range of time. In the case of the columnar database, we only need a few fields – 

symbol

,

price

, and

transaction_date

– in order to complete the query. With a row store, we would gather additional data that was not needed for the query but was still part of the table structure.

Normalization of data also makes updates to some information much more efficient in a row store. If you change an account holder’s address, you simply update the one record in the accounts table. The updated information is available to all transactions completed by that account owner. In the columnar database, since we might store the account information with the transactions of that user, many records might need updating in order update the available address information.

Conclusion

So, which one is right for you? As with so many things, it depends. You can still perform data analysis with a row-based database, but the queries may run slower than they would on a column store. You can record transactions in a column-based model, but the writes may takes longer to complete. In an ideal world, you would have both options available to you, and this is what many companies are doing.

In most cases, the initial write is to a row-based system. We know them, we love them, we’ve worked with them forever. They’re kind of like that odd relative who has some real quirks. We’ve learned the best ways to deal with them.

Then, we write the data (or the relevant parts of the data) to a column based database to allow for fast analytic queries.

Both databases incurred write transactions, and both also likely incur read transactions. Due to the fact that a column-based database has each column’s data in a separate file, it is less than ideal for a “SELECT * FROM…” query, since the request must access numerous files to process the request. Similarly, any query that selects a single or small subset of files will probably perform better in a row store. The column store is awesome for performing aggregation over large volumes of data. Or when you have queries that only need a few fields from a wide table.

It can be tough to decide between the two if you only have one database. But it is more the norm that companies support multiple database platforms for multiple uses. Also, your needs might change over time. The sports car you had when you were single is less than optimal for your current family of five. But, if you could, wouldn’t you want both the sports car and the minivan? This is why we often see both database models in use within a single company.

Dec
10
2016
--

Software is due for a bundling event

Overhead shot of a small group of people, wearing monochromatic colors, pulling at ropes from opposing directions We are approaching a new phase of enterprise software, where every niche of Software-as-a-Service has been filled and cloud companies are being consolidated into larger companies. Markets have a tendency to cycle from bundling to unbundling, and software is due for a bundling event. Read More

Nov
28
2016
--

How to estimate a company’s health without really trying

Conceptual image of a female doctor with a stethoscope Within the past few months, NetSuite, Marketo, LinkedIn, FleetMatics and LogMeIn have each been acquired or merged for a combined value of more than $50 billion. At this rate, public SaaS companies may become an endangered species. Clearly, PE investors and larger technology companies sense opportunity and value. Read More

Nov
25
2016
--

Mobile and enterprise are the keys to VR/AR scale

vr wireframe humans Though PC and console VR are the sexier formats we’re all excited about, is mobile where VR will really scale in the near term? This is a question I’ve been posing to investors and innovators. Read More

Nov
19
2016
--

How data science and rocket science will get humans to Mars

Laptop computer with red ethernet cable forming a rocket, coming out of the back on a plain background President Obama recently re-affirmed America’s commitment to sending a manned mission to Mars. Think your data science challenges are complicated? Imagine the difficulties involved in mining data to understand the health impacts of a trip to Mars. When sending humans “where no one has gone before,” there are a multitude of variables to consider, and NASA is hard at work… Read More

Nov
13
2016
--

How drones will reshape the enterprise

battle-drones Most people associate drones with troops and mad scientists tinkering around in their backyards. Thanks to technological breakthroughs — including longer and safer flights — and new federal guidelines enacted this year, drone use is expanding beyond military and consumer markets and is seeping into the enterprise. Read More

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com