Pages

Showing posts with label Data management. Show all posts
Showing posts with label Data management. Show all posts

2016-01-19

A personal impact of data management

The personal impact of data management.

Some time ago I began feeling odd.

After a few Google searches to get a better idea of what my symptoms could be attributed to we decided to go to the emergency room.

A number of tests later it was decided I needed to have my appendix removed.

Diagram showing the importance and result of w...
Diagram showing the importance and result of well thought out Student Data Management. (Photo credit: Wikipedia)
Throughout this process I was asked at each "station" that I was sent to for simple things like "What brings you in tonight?" or "Why are we testing you?". Simple straightforward ice-breaker questions. As the night wore on and the medication for pain they gave me began to take more of a toll I didn't really think about why they were asking the questions they were asking.

I have been to this particular hospital a few times for various tests, so they already had my insurance and contact information, yet when they sent someone from admissions they had to re-confirm all of my insurance, contact and emergency notification information.

 I said to the lady that it was in the system, since nothing has really changed. The response I received was that they upgraded to a new patient care system in December. They have access to the historical data for records purposes but they have to re-enter the patient information into the new system whenever a previous patient comes in for a current visit.

I was already medicated, so little of the rest of the conversation do I remember.

I basically handed over my license and insurance information and did my best to answer her questions.

However, I do remember this part of the conversation, because this is what I do.

I manage data.

The mental image ran through my mind of some management team discussing the complications of migrating historical data from the legacy system into a new system. (A discussion I have been a part of on numerous occasions.) I could see the person at the head of the table shaking their head and saying:

"It will cost too much money to do a historical migration of everything. We will only migrate enough data from the old system to the new system to keep things going. Any historical patient data is not that important."

I had pretty much heard those exact words from a CIO before.

As I have written before, the historical data you have represents your customers. Ignoring their history is to ignore your own history. What would be the effect, if every time you went in to a bank you had to fill out all of the forms for opening an account?

Each, and every time?


Now I do more with understanding the nuances of what the data represents, and why customers,patients,prospects, or other things that data represents are doing the things they are doing.

 This episode stuck with me for some time, and I hope that we as those that are responsible for data recognize that the data we work with represents real living breathing people.

Historical data is important.

Ignore it at your peril.









2016-01-17

Packaged reports versus integrated reporting.

Data Silo?

Software Packages like SAP, Peoplesoft, or Kronos are dedicated to solving problems. They are full-featured packages that come with plenty of options to meet the needs of their customers.

However a lot of people I speak with about these packages act as if these packages are the only packages in the enterprise.  More often than not, these "all encompassing" packages are not alone in the enterprise; there are other systems with which they need to interact.

A package is not alone

I was reminded of some of my discussions about this topic when I saw a toy commercial recently. They show children playing with just that particular toy set that the advertisement is showing. In reality this one toy set is part of an entire room full of toys.  Cleaning up all of these toys can become a challenge.

If you have ever helped a child find a part of a toy you will understand.

For a toy set if you want to take a car from one set and play with that car on a new race set you simply pick up the car and start playing with it. If it doesn’t quite fit the child will use their imagination to make the car either a giants car, or a tiny robot car looking for other tiny cars in that world.

For an application it’s not quite that simple. If you have a large volume of data in an older application or legacy application, and you want to use that data in your new application you have to have a migration strategy requiring expertise in both applications as well as data manipulation. If these applications use different data repositories, Oracle and the other SQL server, that can make it more complicated.

If you want to use both applications at the same time, during a soft launch, for example, data integration becomes even more vital.

Data needs to be integrated

Constant feedback of performance is a vital business function. Seeing the performance of only one application will limit your vision in other areas. Data Management professionals are the ones ultimately responsible for knowing where all of the “toys” are located and how to get them as well show how they are represented.


Proactive data management planning ensures that some of these problems never emerge. If every week or month or day you have lots of IT people and Business Analysts looking for data, you need to re-think your strategy.Because then you are working for your data, rather than having your data work for you.


Enhanced by Zemanta

2016-01-05

Data Warehouse Maturity

Not all Data Warehouse projects are created equal.


In my years of working in the data management profession, I have worked with a variety of organizations. In the last ten years or so I have worked on many projects that were all called data warehouses. Unfortunately, I can't say that any two have been alike.

Data Warehouse Overview
Data Warehouse Overview (Photo credit: Wikipedia)
Based on my experience, I can say that of all of the organizations I have worked with each fell into at least one of these categories for Data Warehouse maturity. Some organizations I have been able to take from one level to another, but it takes persistence and support from management. 



Level 0 – No formal business intelligence capabilities within the organization  

Reports are ad-hoc feeding data into local client tools managed by analysts.  Multiple people do the same work for both data extraction and report creation.  There is little cross department usage of data.  In addition, change control is not a huge priority since the majority of the analysis is ad-hoc.  The source code used to develop the reports and data transferred is not managed under a revision control system.  Queries are changed easily between runs making historical analysis difficult if not impossible. 

Level 1 – Using add-on tools supplied by vendors

This would include “built-in” data marts provided by vendors for data analytics of a particular application.  This could include, for example, SAP BW, which is a dedicated data mart solution for SAP transactions.  Even though SAP may be rather large, it is nevertheless a single data source with a predefined baseline data mart. 

Level 2 – Departmental data mart(s) with a front-end Business Intelligence tool or an Operational data store feeding common reports.  At this level, dedicated equipment for reporting and analysis is used. 

English: Datamart Architecture Pattern used Bu...
English: Datamart Architecture Pattern used Business Intelligence Reference Architecture used in conjuction with Operational Data Store and Datawarehouse. Supports Reporting , Analytics , Scorecards and Dashboard (Photo credit: Wikipedia)
Data from multiple systems integrated into a common back-end repository.  From a Kimball perspective, this is the foundation of the conformed dimension usage.  The Operational data store can be a copy of the data used by applications, but it can also be an extract of key data from the operational system to be able to provide KPI’s in a single location for multiple departments.

At this level of Data Warehouse Maturity, enough analytical and reporting queries are happening that the load needs to shift to dedicated equipment in order to reduce the load on the production systems.  Analysts will want to be looking at various scenarios that could negatively affect the native use of production applications so offloading this data and the analytical queries onto dedicated equipment becomes a priority for the organization.

This level of maturity requires, at a minimum, some dedicated technical resources to manage the infrastructure of the BI tool(s) and the job processing that controls the movement of data through the various stages that will ultimately make the data available to the BI tool.


This may not be the sole function of these personnel, but organic personnel should be managing this infrastructure.


Cross department usage of data begins here as analysts from different departments use data sourced from one or more systems into a common back-end repository.



Level 3 – ETL tool driven multiple data marts with a front-end business intelligence tool sharing sourced data with meta-data driven from ETL and data modeling tools

At this point in the growth of an organization, the volume of data moving between systems becomes large enough that sophisticated and dedicated tools becomes  a requirement to expedite the data transfer.

 Level 4 – Sophisticated “staging” area using industry best-practices (DW2.0)

A  DW2.0 integration sector called a “staging” area using Kimball terminology. The Presentation layer is generally what Kimball would call a data mart or basic dimensional model.


Level 5 –Multiple Data warehouse environments. (Large international organizations

This would be a really large organization, or one that had grown through the process of acquisitions. In the case of acquisitions, a Data Structure Graph should probably be used to keep analyze how data flows through an organization.


Why does it matter which level an organization is?  How can someone use these levels to affect their business bottom line?

Knowing how fare your organization is in this maturity process will help as you add other infrastructure like Big Data, or other systems. Also to go from one level to another is more of an incremental growth step. Each step takes a lot longer to go. From Level 0 to Level 1 takes a few weeks. From Level 1 to Level 2 takes a few months. Level 2 to 3 takes about 6 months. Level 3 to Level 4 may take up to a year to get everything complete. Level 5 is usually ongoing. Once your organization gets to Level 3 the data warehouse should be at critical mass where more and more people migrate to using it and other systems should be discontinued.


2011-07-25

Datagraphy or Datalogy?

What is the study of data management best practices?

Do data management professionals study Datagraphy, or Datalogy?


A few of the things that a data management professional studies and applies are
  • Tools
    • Data Modeling tools
    • ETL tools
    • Database Management tools
  • Procedures 
    • Bus Matrix development
    • User session facilitation
    • Project feedback and tracking
  • Methodologies 
    • Data Normalization
    • Dimensional Modeling
    • Data Architecture approaches


These, among many others, are applied to the needs of the business. Our application of these best practices make our enterprises more successful.


What should be the suffix of the word that sums up our body of knowledge?

Both "-graphy" and "logy" make sense, but let's look at these suffixes and their meaning.


-graphy

The wiki page for "-graphy"  says: -graphy is the study, art, practice or occupation of... 

The dictionary entry for "-graphy" says -"a process or form of drawing, writing, representing, recording, describing, etc., or an art or science concerned with such a process"


-logy

The wiki page for  "-logy"  says -logy is the study of ( a subject or body of knowledge).

The dictionary entry for  "-logy" says: a combining form used in the names of sciences or bodies of knowledge. 


Data

The key word that we all focus on is data. 

In a previous blog entry, I wrote a review of the DAMA-DMBOK  which is the Data Management Association Data Management Body Of Knowledge. 


Data Management professionals study and contribute to this body of knowledge. As a data guy, I am inclined to study to works of those who have gone before. I want to both learn from their successes and avoid solutions that have been unsuccessful. 


Some of the writings I study are by people like:  Dan LinstedtLen Silverston, Bill Inmon, Ralph Kimball, Karen Lopez, William Mcknight and many others. 

I have seen first hand what happens to a project when expertise from the body of knowledge produced by these professionals has been discarded. It is not pretty. 


Why do I study these particular authors? These folks share their experiences. When I face an intricate problem, I research some of their writings to see what they have done. Some tidbit of expertise they have written about has shed light on many problem I have faced, helping me to find the solution that much sooner.


When I follow their expertise my solutions may still be unique, but the solutions fit into patterns that have already been faced. I am standing on the shoulders of giants when I heed their advice. 


When I am forced to ignore their advice, I struggle, fight and do battle with problems that either should not be solved or certainly not be solved in the manner in which I am forced to solve them. 


Should the study of and contribution to the body of knowledge of data management be called data-graphy or data-logy? 


Datagraphy

The term Datagraphy sums up the study of the data management body of knowledge succintly. 

I refer back to the dictionary definition of the suffix "-graphy": "a process or form of drawing, writing, representing, recording, describing, etc., or an art or science concerned with such a process"

Data is recorded, described, written down,written about, represented (in many ways) and used as a source for many drawings and graphical representations. 


What do you think? I will certainly be using Datagraphy.
Enhanced by Zemanta

2011-05-27

Analytical Skills?

One particular skill listed on many job descriptions I have seen gives me more questions than answers.

The skill they would like the candidate to have is listed as : Analytical Skills


How is this quantified? How do you know that someone has analytical skills?


Wiki defines Analysis as the ability to break down a complex topic or substance into smaller parts to gain a better understanding of it.

To what level of detail does this particular job need Analytical Skills? Will the candidate need to analyze the Enterprise accounting functions? Does it need to be the analysis of the physical server infrastructure? Does it need to be organizational layout? Will the candidate be analyzing forensic evidence?

Analytical skill is such a broad topic, and once a person does their analysis what happens to it? Is this a position that has the ability to not just analyze data, but also act on it? Is the analysis required in the position for recommendation purposes, educational purposes, or will the candidate be making decisions based on data provided by others?

Many people have analytical skills, but do they have good analytical skills? Do decision makers listen to and follow their recommendations after an analysis is done?

Data Management professionals are constantly analyzing data. The raw data can represent many diverse topics. Some of the key topics for analysis that come to mind are People, Processes and Things.

People

The analysis of people, their motivations, and their interactions is covered by subjects like anthropology, psychology, sociology and other behavioral sciences. Some people are naturally gifted people readers and can understand others with limited formal training. The analysis of people is useful to many groups within an organization, human resources, marketing, sales, even executive leadership.

Process

There are many types of processes in our lives, a process for getting a drivers license, getting married, fulfilling a product order, shipping a product, and many others. Understanding and recommending improvements to the nature of the processes that we interact with on a daily basis can be very valuable.

Things


Things can be companies, human languages, computer languages, web pages, corporate ledgers, computers, cars, religions, money, inventories, nature. Every "thing" can be studied and analyzed. The more we understand things the more data we generate about those things in order to contribute to human knowledge, self knowledge or our corporate enterprise.

Data

All things that are analyzed have one thing in common. Data. In all analysis data is what is collected, stored, manipulated, reported, recommended and decided upon.

There are best-practices for data management that can assist every type of analysis of every subject. "Pure" analytical skill is seldom used in a vacuum. Databases store data, operational systems collect data, a data warehouse helps in the correlation of data amongst multiple systems that are gathering data. Data Management by its very nature is an analytical skill.

When I see a position for a data management professional that requires analytical skills, I still find it humorous. Because the analytical skills that we can provide in both our own analysis and in guiding the analysis of others should go without saying.
Enhanced by Zemanta

2011-02-17

Thought leadership

You have to be a thought leader in order to recognize one.

I hear the term thought leader bestowed upon people occasionally. I have even bestowed this term on some people that I consider to be extremely knowledgeable about building data warehouse systems.

The wealth of information for data management best practices continues to grow. Thought leaders can publish knowledge about solving a particular problem in a variety of forums now: blogs, books, articles, and even research papers. The sheer volume of information about the "best practices" is almost intimidating.

The ability to take in all of the information about best practices for a subject area, apply it to the situation at hand, consolidating the recommendations from multiple sources as well as ignoring those recommendations that are not applicable make you a thought leader. Google provides a way of finding a site that answers a particular question. If a person does not ask the correct question, Google does not provide a good answer. Once Google finds a particular answer to a keyword query you have to apply that answer to your particular situation.

Let us take a specific example.

The question should not be:

What is the best way to build A data warehouse?

The question should be:

What is the best way to build THIS data warehouse?

Even something as simple as learning a how to apply a new SQL trick that you learned to a specific problem you are working on shows the application of this knowledge. Best practices can be abstract, or even theoretical. When you can take recommendations from many sources and apply their expertise to your specific problem you have taken a big step.

This can apply to many other professional areas.  SEO, Business Analysis, Business Process Re-engineering, ETL development,Resume writing, Financial Analysis, Online Marketing,  etc...


If you can study multiple sources and apply their recommendations or findings to your own situation, you become a thought leader.

You become a recognized thought leader when you write about it. 




Enhanced by Zemanta

2010-12-27

DAMA-DMBOK book review

Diagram showing the importance and result of w...Image via Wikipedia
“What do you do?”

I am asked this question frequently. Family members, church friends, even recruiters and coworkers sometimes ask this question.

Depending on the audience, I will say something like “work with computers”, or “I’m a DBA.” or “I’m a database developer.”

Dr. Richard Feynman once said: “If you can't explain something to a first year student, then you haven't really understood it.”

The DAMA – Data Management Body of Knowledge is a work that attempts to document and formalize the definition of the Data Management profession.

According to the book, a Data Management Professional is responsible for the planning, controlling and delivery of data and information assets.


The thing that impressed me the most is that it brought together so many formal definitions of many various concepts that I work with on a daily basis. Whole books can, indeed have, been written on each component of data management touched on in the body of knowledge. One of the values of this book is the bibliography. If one were to acquire every book referenced in this work they would have an impressive library of data management knowledge.

Another thing that was impressive to me is this book advocates the role of the Data Management Executive. The Data Management Executive is defined as: “The highest-level manager of Data Management Services organizations in an IT department. The DM Executive reports to the CIO and is the manager most directly responsible for data management, including coordinating data governance and data stewardship activities, overseeing data management projects and supervising data management professionals. May be a manager, director, AVP or VP.” I have worked with and in many organizations; very few actually had an “official” data management executive. As a result, data movement into and out of the organization has been something of a haphazard process. Each project that required movement of data was approached differently. If a single official point of contact for all data management activities existed, then these projects could have been more streamlined to fit into an overarching design for the enterprise as a whole.

Each chapter covers a different aspect of the overall Data Management Profession.  The first chapter gives an overview of why data is a corporate asset. The definition of data as a corporate asset is the foundation of all data management activities. Focusing on data as an asset first, then the follow on activities discussed in the major component chapters are seen as value-add activities.
 This picture illustrate the Data Architecture ...Image via Wikipedia
The major components of Data Management covered by the chapters and the definitions the DMBOK provides are:


Data Governance: The exercise of authority and control (planning, monitoring and enforcement) over the management of data assets. The chapter gives an overview of the data governance function and how it impacts all of the other functions. Data Governance is the foundation for the other functions.

Data Architecture: An integrated set of specifications artifacts used to define data requirements, guide interaction and control of data assets, and align data investments with business strategy.

Data Development: The subset of project activities within the system development lifecycle (SDLC) focused on defining data requirements, designing the data solution components and implementing these components.

Data Operations Management: The development, maintenance and support of structured data to maximize the value of the data resources to the enterprise. Data operations management includes two sub-functions: database support and data technology management.

Data Security Management: The planning, development and execution of security policies and procedures to provide proper authentication, authorization, access and auditing of data and information assets.

Reference and Master Data Management: The ongoing reconciliation and maintenance of reference data and master data.

Data Warehouse and Business Intelligence Management: This is a combination of two primary components. The first is an integrated decision support database. The second is the related software programs used to collect, cleanse, transform, and store data from a variety of operational and external sources. Both of these parts combined to support historical, analytical and business intelligence requirements.

Document and Content Management: The control over capture, storage, access, and use of data and information stored outside relational databases. Document and Content Management focuses on integrity and access. Therefore, it is roughly equivalent to data operations management for relational databases.

Meta-data Management: The set of process that ensure proper creation, storage, integration and control to support associated usage of meta-data.

Data Quality Management: A critical support process in organizational change management. Changing business focus, corporate business integration strategies, and mergers, acquisitions, and partnering can mandate that the IT function blend data sources, create gold data copies, retrospectively populate data or integrate data. The goals of interoperability with legacy or B2B systems need the support of a DQM program.


The last chapter covers Professional Development, ethics, and how DAMA( Data Management International) dama provides a professional society body or guild for the communal support of information and data management professionals.

Overall this is an outstanding book for defining the roles associated with data management. While it is light on details for implementing the programs, processes and projects that it defines, it is nevertheless a great book for creating a common vocabulary amongst professionals who work day-to-day in the data management profession.

The more we, as data management professionals, communicate consistently with business users, executives, and the public about what we do the better it will be for all of us when one of us is asked “what we do”.

My answer now is I am a Data Management Professional. I can assist you with better understanding, delivery, analysis, security and integrity of your data. 



Enhanced by Zemanta

2010-12-01

Data as an Enterprise Asset

From the wiki an Asset is: "Anything tangible or intangible that is capable of being owned or controlled to produce value and that is held to have positive economic value is considered an asset"

Data is the most valuable Enterprise Asset's in existence. The recent release of documents on the Wikileaks site is a prime example of this. What will be the final cost associated with the release of these documents? How many man-hours will be devoted to changing procedures, implementing new security protocols, trying to recover loss of face by many government agencies?


Your address book


What would happen if you were to lose your phone?

You would just replace it right?

What about your address book?

How many people keep their contacts in multiple locations for "safekeeping"?

You want to make sure that you keep your contacts regardless of what happens to your phone, right. Data Management professionals feel the same way about the data that they safeguard.

The CEO view

It is 11:43 p.m. on a Friday night. The alcohol from the dinner meeting with investors won’t wear off for a few more hours. You should be fine for the 7:12 a.m. tee time with the next group of potential clients. When the phone rings you just curse and pick it up.

“What!” you yell into the phone.

“Hey boss “, you hear the head of your IT department.

“Listen; there is no easy way to say this. In the storm that we had earlier tonight, we took a handful of lightning strikes and had a tornado touch down on the building itself. The lightning strikes then caused a fire that wasn’t caught until it was too late. The building is pretty much destroyed.

We have already updated DNS to our DR site. Some of the DBA's and server admins are on the way there. Our main network guy is unavailable since he is out of town. We are supposed to have our backup tapes there in a few hours. The server guys will get our servers back up, the DBA's will restore the databases and validate where we are with the data."

What do you do?

If you trust the DR plan and your DBA's, then you can go back to sleep.

Would you sleep well?

How valuable is your data now that you don't know whether you have it or not?

Valuating your data

One way to determine the value of your data is to identify the direct and indirect business benefits derived from use of the data. Another way is to calculate the cost of its loss; what would be the impact of not having the current quality level of your data or the amount of data you have?

What if you only lost a years' worth of data?

What change to revenue would occur?

What would be the cost to recover it? Man-hours, potentially consultant hours as you hire outside expertise if necessary would factor in to the costs.

Data Management Professionals protect your Assets


Data Management Professionals are the ones that protect your data assets. By protecting and safeguarding your data assets, they are protecting and safeguarding the enterprise itself.
Enhanced by Zemanta