I’ll be presenting a fun session at the main event (Cats, Facebook, and Online Dating with Microsoft BI), and doing a full day training session this Friday (6/21). If you are interested in learning more about SSIS, DQS, and MDS, please see the Prodata site for details.

Hope to see you there!

Think of your data as “water”, and your knowledge base as a “water filter”… as your filter improves in quality (you improve your knowledge base), the water (data) you run through it gets cleaner.

I usually recommend storing the date value of when the data was processed with DQS. This could be a column value, if you want to track things at the row level, or a entry in a separate file/table, if you’re tracking it at the table/DB level. (I describe this pattern in detail in the SSIS Design Patterns book)

While there is no official notification system or API to know when a DQS knowledge base has been updated, the DQS database does contain a PUBLISH_DATE field that you can query. This won’t tell you which individual domains were touched, or what was actually modified, but given that you probably won’t be publishing a new version of your knowledge base without a good reason, it should be good enough for most automated data cleansing scenarios.

Query the Publish Date for a DQS Knowledge Base

We’re interested in the [dbo].[A_KNOWLEDGEBASE] table in the [DQS_MAIN] database. This table lists all of the knowledge bases in your system, as well as any active projects. You can use the following query to get the last time the knowledge base was updated (replace the [Name] clause with the name of your KB).

SELECT [ID]
      ,[NAME]
      ,[PUBLISHED_BY]
      ,[PUBLISH_DATE]
  FROM [dbo].[A_KNOWLEDGEBASE]
 WHERE [TYPE] = 1 -- knowledge bases only (no projects)
   AND [NAME] = N'DQS DATA'

Staging Tables and Stored Procedures

The new entity based staging tables are an excellent feature in MDS 2012, and make it very easy to bulk load into MDS from SSIS. If you take a look at the SQL database used by your MDS instance, you’ll see at least one table in the stg schema for each entity. For this example I’ve created a Suppliers entity and I see a matching table called [stg].[Suppliers_Leaf]. If your entity is using hierarchies, you will have three staging tables (see BOL for details). If we expand the columns, we’ll see all of the attributes have their own columns, as well as some system columns that every staging table will have.

Each staging table will also have a stored procedure that is used to tell MDS that new data is ready to load. Details of the arguments can be found in BOL.

Import Columns

To load into this table from SSIS, our data flow will need to do the following:

• Set a value for ImportType (see below)
• Set a value for BatchTag
• Map the column values in the data flow to the appropriate attribute columns

See the Leaf Member Staging Table BOL entry for details on the remaining system columns. If your Code value isn’t set to be generated automatically, then you’d also need to specify it in your data flow. Otherwise, the default fields can be safely ignored when we’re bulk importing.

The BatchTag column is used as an identifier in the UI – it can be any string value, as long as it’s unique (and under 50 characters).

MDS uses the same staging table for creating, updating and deleting entities. The ImportType column indicates which action you want to perform. The possible values are listed in the table below.

Value	Description
0	Create new members. Replace existing MDS data with staged data, but only if the staged data is not NULL. NULL values are ignored. To change a string attribute value to NULL, set it ~NULL~. To change a number attribute value to NULL, set it to -98765432101234567890. To change a datetime attribute value to NULL, set it to 5555-11-22T12:34:56.
1	Create new members only. Any updates to existing MDS data fail.
2	Create new members. Replace existing MDS data with staged data. If you import NULL values, they will overwrite existing MDS values.
3	Deactivate the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are maintained but no longer available in the UI. If the member is used as a domain-based attribute value of another member, the deactivation will fail. See ImportType 5 for an alternative.
4	Permanently delete the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are permanently deleted. If the member is used as a domain-based attribute value of another member, the deletion will fail. See ImportType 6 for an alternative.
5	Deactivate the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are maintained but no longer available in the UI. If the member is used as a domain-based attribute value of other members, the related values will be set to NULL. ImportType 5 is for leaf members only.
6	Permanently delete the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are permanently deleted. If the member is used as a domain-based attribute value of other members, the related values will be set to NULL. ImportType 6 is for leaf members only.

When you are bulk loading data into MDS, you’ll use 0, 1 or 2 as the ImportType. To summarize the different modes:

• Use 0 or 2 when you are adding new members and/or updating existing ones (i.e. doing a merge)
• The difference between 0 and 2 is the way they handle NULLs when updating an existing member. With 0, NULL values are ignored (and require special handling if you actually want to set a NULL value). With 2, all values are replaced, even when the values are NULL.
• Use 1 when you are only inserting new members. If you are specifying a code, then a duplicate value will cause the import to fail.

Package Design

You control flow will have at least two tasks:

1. A Data Flow Task that loads your incoming data into the MDS staging table for your entity
2. An Execute SQL Task which runs the staging table’s stored procedure which tells MDS to start processing the batch

Your data flow will have (at least) three steps:

1. Read the values you want to load into MDS
2. Add the BatchTag and ImportType column values (using a derived column transform)
3. Load into the MDS staging table

As noted above, in your OLE DB Destination you’ll need to map your data flow columns to your member attributes (including Code if it’s not auto-generated), the BatchTag value (which can be automatically generated via expression), and the ImportType.

After the Data Flow, you’ll run the staging table stored procedure.

EXEC [stg].[udp_Suppliers_Leaf] ?, ?, ?

The first three parameters are required:

1. The version name (i.e. VERSION_1)
2. Whether this operation should be logged as an MDS transaction (i.e. do you want to record the change history, and make the change reversible?)
3. The BatchTag value that you specified in your data flow

Additional resources:

• SQL 2012 Master Data Services – Entity Based Staging (video)
• Importing Data (Master Data Services)
• Staging Process Errors (Master Data Services)

I want to extend a big thank you to everyone who attended, and to all the great feedback we got. It can be tough doing a 400 level SQL session at TechEd, and while I could see some people’s heads spinning, it sounded like most people were able to learn something new.

The TechEd team picked an excellent preview picture for the session (below). It comes from Matthew’s intro – you’ll have to watch the video to see how he worked a picture of kittens into a 400 level SSIS session.

If you didn’t already know, Channel 9 has many TechEd presentations available online. You can see recordings of my previous events on my speaker page, and Matthew’s as well.

Please come see me at one (or more) of the following events:

• 05/22 – Webcast: Developing Extensions for SSIS
• 05/27 – Montreal User Group
• 05/30 – DevTeach Conference in Toronto, Ontario
• 05/31 – EIM Training in Toronto, Ontario
• 06/03 – TechEd North America 2013 in New Orleans
• 06/21 – Enterprise Information Management all day training session in Dublin, Ireland
• 06/22 – SQL Saturday #229 | Dublin, Ireland
• 06/25 – TechEd Europe 2013 in Madrid, Spain

Integration Services Deployment Wizard

Yes, this is the primary UI tool for SSIS project deployment, but it can also be run silently on the command line. When you run through the wizard, the Review page will actually list all of the parameters you need to do the same deployment from the command line.

Run ISDeploymentWizard.exe /? from a command prompt, and you’ll see the full list of arguments it supports.

SSIS Catalog T-SQL API

The SSISDB [catalog] schema has a number of public stored procedures, including one that can be used for deployment. We even provide samples on how to use it in Books Online.

DECLARE @ProjectBinary as varbinary(max)
DECLARE @operation_id as bigint
Set @ProjectBinary = (SELECT * FROM OPENROWSET(BULK 'MyProj.ispac', SINGLE_BLOB) as BinaryData)

Exec catalog.deploy_project @folder_name = 'MyFolder', @project_name = 'MyProj', @Project_Stream = @ProjectBinary, @operation_id = @operation_id out

PowerShell

All SSIS Catalog operations can be automated through PowerShell. I previously blogged about a deployment script I use to setup my demos.

Custom Utility

The SSIS Catalog management object model (MOM) exposes a set of SMO classes you can use to code your own catalog utilities. You’ll want to use the CatalogFolder.DeployProject method to do the actual deployment. If SMO or .NET isn’t your thing, you can also code a custom utility which interacts directly with the T-SQL API.

I put this together for an SSIS performance presentation I did a while ago, and thought it warranted its own post.

In addition to looking for appropriate Design Patterns, I have a checklist of things to look for when trying to optimize an SSIS Data Flow. This list is by no means extensive, but it contains some of the more common causes of performance issues in the packages I’ve worked with.

• Can you remove the OLE DB Command transform?

OLE DB command operates on a row by row basis. Stage the data and perform the operation in a batch mode using the Execute SQL Task instead.

• Can you replace Upsert logic with a MERGE statement?

Learn MERGE, and love it.

• Do you need those Sort transforms in there?

Sometimes you do. Most of the time you don’t. If you do need them (i.e. you are loading from a flat file, and need to do a merge), considering maxing out the DefaultBufferMaxRows setting so you can sort more data in memory (sort will keep at most 5 active buffers in memory before it starts swapping to disk).

• Lookup Transforms
• Can the lookup be avoided?
• Are you using the right cache mode?
• Should you use a Cache Connection Manager?

Lots of related lookup posts can be found here.

Have fun optimizing!

Andy has a great blog series about creating custom SSIS Tasks, and his company (Linchpin People) recently released an SSIS Event Task. I plan on expanding on the content in Andy’s blog posts a little bit, and talk about some of the key things you need to know when building an SSIS Data Flow component.

The abstract for the talk is available below, and registration page can be found here.

Join Matt Masson and Andy Leonard for a discussion and demonstrations on extending SSIS with custom tasks and data flow components. This code heavy session walks you through the creation of a custom SQL Server Integration Services (SSIS) Task, and a custom Data Flow component. If you’ve ever thought of customizing SSIS functionality beyond what you can do with scripting, this talk is for you! All examples will be in coded in C#, and made available following the session.

I’m not sure when it started, but for a while now I’ve found it hard to just sit and watch TV or a movie with my family. I feel like it doesn’t occupy my mind enough, and I need to be doing something else at the same time. My Surface has come in really handy here – usually if I’m watching something, I’ll be doing something with it at the same time. I’m sure I’m not alone in this.

There’s a lot of stuff going on within my organization, and I like to have a general idea of what everyone is doing. Recently I’ve been spending a lot of time reviewing specs, wireframes and power point presentations from other teams. We make heavy use of SharePoint internally, both the corporate and cloud versions, and that’s generally where I go to find the docs I’m looking for.

I thought I could make good use of my “quality family time in front of the TV” by combining these two activities. However, when I’m at home, accessing the documents I need from my tablet can be tricky. When they are stored in SharePoint Online, I can get to them through my browser, but getting to documents in our internal SharePoint sites is harder because they are behind the corporate firewall. I started searching around for solutions, and came across a new service called the Windows Azure Application Gateway. Microsoft has ways for employees to connect to the corporate VPN on a Surface (using our smartcards), but this new service seemed like a light weight approach and looked interesting.

Windows Azure Application Gateway

The interoperability @ Microsoft blog had a great blog post about this service:

For obvious security reasons, connecting to resources that are behind a firewall usually requires a fairly complex infrastructure such as VPN (Virtual Private Network). The AppGateway demo app is designed to make the process simpler yet highly secured. The mobile app connects to a service on Windows Azure that acts as the proxy to an agent that is running inside the network behind the firewall. Using the Windows Azure Authentication service, the proxy can establish a trusted connection to the agent so that the application on the mobile device can browse web sites that would not be normally accessible outside the corporate network.

The site has an easy to understand explanation of what the service does, and how it works. It was actually what caught my eye… I really like the graphic they use. I call him Mr. Firewall.

The usage scenario they described fit what I was trying to do – there were corpnet resources I wanted to access, but couldn’t access them from home.

Many companies will provide a way to connect to their internal corporate network (for example, DirectAccess or connecting through a VPN). These solutions tend to work fine from a laptop, but might not always work so well for a tablet or phone. Surface RT supports VPN connections, but many corporate networks require some form of custom VPN client or smartcard solution that might make it harder (or more awkward) to use. If your company doesn’t offer a VPN solution, or you’re in a reverse scenario where you want to access your home network while you’re at work, then you might be out of luck.

Their walkthrough graphics got me hooked, and I decided to try it out.

Usage

To use the service, you need to create login with either Windows Azure Active Directory (WAAD) or Office 365. (You can sign up to WAAD for free).

Once you have a login, you can download the connector and install it on your corporate machine (or your home machine, if you’re trying to create a bridge from work to home).

After install, you’ll see an Application Gateway icon in your notification area, and a browser pops up asking you to login and register your connector.

After registration, you get a webpage dashboard on your connector machine, with live tiles.

I like the detailed status window that lets you test your connectivity:

Once your connector is setup, you can use it to access web sites in a few different ways. There are custom iOS and Android client apps available, and on a windows OS (Windows Phone, Surface, PC) you access the Web Client directly using your browser. Here is a screenshot from my Surface RT:

Click on the name of your connector, and it brings up a web browser interface. From here you can enter intranet URLs. The data from the site gets passed back through your connector machine and rendered on your device. It’s as if you were directly connected to your network using a VPN.

Conclusion

I’ve just started using the service, but it seems to work great so far. I’m able to browse internal SharePoint sites, and view (and edit) Word/PowerPoint files on my Surface. It hasn’t been out long (it’s still in Preview), so I expect there will be further improvements before it becomes generally available.

A more detailed explanation of how the service works can be found here, and it looks the source code for their Android demo app is available on Github.

Try it out and let the team know what you think!

• ConnectionDescription
• ConnectionDisplayName

These must be public, static string properties on your LocalizationType class.

The code looks like this:

[DtsConnection(ConnectionType = "MYCONNECTION", LocalizationType = typeof(Strings))]
    public class MyConnectionManager : ConnectionManagerBase
    {
    }