Wednesday, September 4, 2013

SSIS passing DATE datatype into Exe SQL Task via Store Procedure

This is a really simple question:

There is one store procedure created on production COS US, now I want to call it via SSIS.

DECLARE @retailerID INT
DECLARE @date      DATE
DECLARE @column_empty  BIT
DECLARE @noData       BIT
DECLARE @cnt        INT
DECLARE @uploads    INT 
DECLARE @exists     BIT
DECLARE @sp_status     INT

SET @retailerid = 27
SET @date  = CONVERT(DATE,'20130903')

  
EXEC @sp_status = dbo.PopulateCostoUS_TransactionFee  @retailerID  ,  @date , @column_empty  OUTPUT, @cnt  OUTPUT, @uploads  OUTPUT,@noData     OUTPUT

SELECT    @retailerID as retailer    ,DATEADD(day,-1,@date)  as [date], @column_empty as column_empty, @cnt as cnt, @uploads as uploads ,@noData as NoDataFlag

 

HOW CAN I DO IT? It’s causing a lot of pains now…????

 

This BLOG saved my life

 

http://blog.hoegaerden.be/2011/06/11/passing-a-datetime-parameter-into-the-execute-sql-task/

 

 

Basically, if the package variable is using DATETIME type , then, in the Parameter mapping section, the data type should be DATE which  consists of year, month, day, hour, minute, seconds, and fractional seconds. The fractional seconds have a fixed scale of 7 digits.

if the SP is using INT and the pacakge variables that’s going to be passed to this parameter is INT32 , then parameter mapping should be LONG, instead of short

In expression, if you want to show the value of the variable, use double quote “, if you want to put the value under the single quote , use +’” @val +”’

Friday, July 26, 2013

Setup ForceCommitTimeout property to SSAS Cube

 

Last blog has introduced a way to kill long running query inside SSAS database. The more elegant way is to let it timeout automatically instead of manual

intervention.

Check this blog

 

http://geekswithblogs.net/darrengosbell/archive/2007/04/24/SSAS-Processing-ForceCommitTimeout-and-quotthe-operation-has-been-cancelledquot.aspx

how to find costly CUBE query and kill it

 

often times, you will find that SSAS server gets super busy because of weird request from CUBE client (proclarity, excel, report builider ), as the SSAS Administrator,

you need to find the culprit and kill . Here is how to do so

1. Find it

 

<Discover xmlns="urn:schemas-microsoft-com:xml-analysis">
  <RequestType>DISCOVER_SESSIONS</RequestType>
  <Restrictions>
    <RestrictionList />
  </Restrictions>
  <Properties>
    <PropertyList />
  </Properties>
</Discover>

 

2. Kill it

 

<Cancel xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">

<SPID>121672</SPID>

<CancelAssociated>1</CancelAssociated>

</Cancel>

 

Now the challenge is to find out the culprit as step 1 will return a list of SPIDs, how do you sift through???

Thursday, July 25, 2013

When perform CUBE sync , error pops up

Similar to below:

 

Database synchronization started.

The physical TCP/IP connection failed: An existing connection was forcibly closed by the remote host. 

The peer prematurely closed the connection.

Backup and restore errors: An error occurred while synchronizing the 'AdventureWorks' database.

An error was encountered in the transport layer.

Database synchronization failed.

 

 

Solution:

 

  • Logon to SSAS "Destination server" (i.e. the destination server on which you want to Synchronize SSAS database from source server).
  • Open "Computer Management" (you can open "Computer Management" simply by typing shortcut "compmgmt.msc" in the "Run" window).
  • Under "Computer Management", expand "Services and Applications" node and then select "Services" node. 
  • Under Services panel, right click on "SQL Server Analysis Services(MSSQLSERVER)" and select Properties.
  • Under Properties window, click on "Log On" tab and select option "This account" instead of "Local System account". Mention the credentials of the active directory user who has admin rights (i.e username and password) and restart the services.

 

http://aniruddhathengadi.blogspot.ca/2011/09/error-while-synchronizing-ssas-database.html

Wednesday, July 24, 2013

A quick overview of SSAS for DBAs

 

http://dba.stackexchange.com/questions/14600/what-does-a-dba-have-to-know-about-ssas

So, you're a SQL Server DBA and you've just inherited some cubes out of the blue to manage. A quick crash course on SSAS administration seems to be in order.

From an administrative point of view, SSAS is a fairly straightforward, if resource hungry application. It's way simpler than a DBMS platform, although different in several ways. Also, there are a few administrative tasks such as usage based optimisation specific to SSAS that you might have to get your head around.

Configuration

All config data lives in a file called msdmsrv.ini. It's an XML file. If you tinker with the config through SSMS (Connect to the server, right click on the server, select properties) you can set options that will break the server on start up. Take a copy of msmdsrv.ini before you play with anything.

Important parameters

Memory: SSAS is a biblical memory hog. It likes 64 bit builds if possible, and lots of memory. The parameters 'Memory\LowMemoryLimit' and 'Memory\HighMemoryLimit' control memory usage policies. LowMemoryLimit is not a minimum memory allocation. It is a threshold where SSAS considers the system is low on memory and starts flushing stuff out of its caches. HighMemoryLimit is the absolute maximum it will use.

Note that SSAS stores its data in files (lots of files - it doesn't have any mechanism equivalent to filegroups), so it makes a lot of use of O/S file system caching for those files. Note that the default for these limits is about 65% and 80% of the machine's memory respectively, so if you want the OLAP server to coexist with a SQL Server instance you will need to turn those down so it isn't fighting over memory with the database server.

Directories: Five parameters of interest cover this: DataDir, AllowedBrowsingFolders, BackupDir, LogDir and TempDir. DataDir and AllowedBrowsingFolders are the most important.

  • AllowedBrowsingFolders affects the list of folders that the OLAP server will put its data files in. Anything with a user interface (e.g. the deployment wizard) will restrict your options to the list in AllowedBrowsingFolders. The value is a pipe ('|') delimited list of directories.

  • DataDir is the default path for files. If you intend to partition the cube over multiple volumes you will need to configure AllowedBrowsingFolders appropriately.

  • LogDir is where the server places various log files including flight recorder and query logs. The flight recorder log is used for troubleshooting and the OLAP query log is used for usage based optimisation (more on this later).

  • TempDir is a location for temporary files that SSAS creates while processing. If you are processing large data volumes and having performance issues, you may benefit from shifting this off onto another volume from the data.

  • BackupDir is what it says on the tin.

Miscellaneous: A few miscellaneous parameters may also be of interest. A couple of sets that you may have to tweak are:

  • DefaultMaxDrillthroughRows: This limits the size of drillthrough row sets. You may need to fiddle this to allow more.

  • Threads/Timeouts: You may need to adjust these. I've never had to bother.

Those are the basics. You may need to tweak the others for specific reasons, but you can do your homework on that.

A reference guide for the SSAS server properties can be found here.

Operations

Deployment: You can compile a project in BIDS and get a set of files that can be deployed with thedeployment wizard. You may need to adjust file paths for partitions and a few other things.

Programmed and batched admin tasks: Commands are issued to SSAS through a web service API called XML/A. Microsoft provides an interactive tool for issuing MDX and XML/A connands. If you have to embed MDX in an XML/A command, watch out for the need to use XML escapes such as &amp;. This is not an issue with the MDX editor and query tool in SSMS.

Offline jobs can be done through various SSIS cube processing tasks, a command line utility calledascmd.exe or a .Net API called AMO. You can also get various powershell tools and suchlike.ascmd.exe takes an XML/A file and posts it to the server. If you have to frig the file programatically you might be better off working with little .Net tools rather than trying to manipulate XML files from a .cmd script.

The operations guide goes into this in more detail.

Security

Security on SSAS is fairly simplistic. It has a global 'Server' role that has administrative permissions across the whole system. Unfortunately you need 'Server' to create databases, so it's quite likely you will need to grant it to developers on any development OLAP servers you have.

Other security can only be applied to individual cube schemas. You can grant permissions to read, process, drillthrough, writeback and suchlike on individual items to roles within a schema. Roles on OLAP schemas can be defined within BIDS and are deployed with the cube. AD groups or users can be assigned to those roles through SSMS.

An example of how to programatically manage role membership can be seen here.

Usage-based optimisation

As a DBA you might well get involved in this one, but first a little backgrounder about the physical storage. SSAS works by calculating and persisting pre-built aggregates along with the base data. If a query can be satisfied by hitting an aggregate the OLAP server will use this in preference to the base data, as the aggregate will involve much less I/O and therefore it will be quicker to retrieve the data.

However, you have to work out which aggregates to calculate (i.e. what combinations of dimension attributes to generate rollups for). BIDS has a tool that will take a guess at this and generate some for you. Some tools, such as BIDS helper will also allow you to manually edit the aggregates.

Usage based optimisation works by taking a log of the actual queries issued to the server, and then using that log to work out a set of aggregates that would be optimal for those queries. As a DBA, you may set up OLAP query logging to capture this data, and then run the optimisation on the cube. A blurb about setting up the query log can be found here.

The tool for doing this is called the 'Usage based optimisation wizard.' This lives in SSMS, and can be found by opening the partition int the explorer and selecting 'Usage Based Optimisation' off the right-click menu.

The performance guide goes into tuning in more detail.

MDX

MDX looks a bit like SQL but works very differently. A treatise on 'MDX for SQL programmers' is an entire topic in its own right. I suggest reading through some of the tutorials on it and/or getting a book on it. Plus, the friendly folks here at dba.se can help with questions if you have any.

MDX doesn't have any concept of filtering rows. The language has a whole lot of set operations to work out what to display on the various axes of the query1, plus 'SELECT'. You can use with statements that look a bit like CTEs to define measures and sets.

Some introductory MDX programming resources can be found here and here (this one is quite old and very long winded). There are also a few good books on the subject; this SO question has quite a bit of fan-out to SSAS resources.

1 Although it does have non-empty operators that restrict the results to combinations that actually have data. Most MDX queries consist of defining what slices to show on the axes, and non-empty operators will be necessary to avoid queries returning combinatoric level quantities of empty cells.

Friday, July 19, 2013

Logging failed when trying to process cube using default impersonation mode

Problem: When I was trying to process the cube whose source data is on staging .18 and target deployment server is localservice, BIDS reported that service account used

can not access the source.  The service account is the local service account

 

Solution: Go to Control panel on local machine –> services-> find out SSAS service-> Log On tab  and change “This account ” to my windows credentials with passwd

Tuesday, July 16, 2013

How to parse XML file with multiple namespaces

 


Steps to parse XML files with multiple namespaces in SSIS

1. Get rid of multiple namespaces

follow the thread below, bascially apply XSLT to regenerate an XML

http://social.msdn.microsoft.com/Forums/sqlserver/en-US/2fde216c-01ab-42a5-8b43-52f5f42d35bc/multiple-namespaces-in-an-xml-document

2. Once cleaned XML has been created, follow the thread below to parse only required information using XSLT

http://blog.hoegaerden.be/2011/04/20/loading-complex-xml-using-ssis/

3. The output is the CSV file which can be inserted into tables easily

Monday, July 15, 2013

OLE DB error: OLE DB or ODBC error: Login failed for user 'NT AUTHORITY\NETWORK SERVICE'.; 28000; Cannot open database "DW_PROD" requested by the login. The login failed.; 42000.

 

Reason:   this user 'NT AUTHORITY\NETWORK SERVICE' does not have read access to DW_PROD

Solution:  Connect to the server via SSMS. in Security section, create this user if doesn’t exist.

Then go to login properties –> User Mapping and choose the database you want to grant permission, grant db_datareader role

Wednesday, June 26, 2013

Permission denided on sp_send_dbmail

I’ve come across this issue where an dbmail task failed inside ssis

After troubleshooting, it turns out the credentails inside the connection manager was not in the msdb db and not a member of databasemailuserRole

Detailed info below:

Link: http://msdn.microsoft.com/en-us/library/ms188719(v=sql.105).aspx

This topic describes how to troubleshoot an error message stating that the user attempting to send Database Mail does not have permission to execute sp_send_dbmail.

The error text is:

EXECUTE permission denied on object 'sp_send_dbmail', 
database 'msdb', schema 'dbo'.

To send Database mail, users must be a user in the msdb database and a member of the DatabaseMailUserRole database role in the msdb database. To add msdb users or groups to this role use SQL Server Management Studio or execute the following statement for the user or role that needs to send Database Mail.

EXEC msdb.dbo.sp_addrolemember @rolename = 'DatabaseMailUserRole'
    ,@membername = '<user or role name>';
GO
 
 

Monday, June 24, 2013

SSIS package template

SSIS package template

http://www.mssqltips.com/sqlservertip/2841/creating-ssis-package-templates-for-reusability/

 

Once you are done with creating the basic structure of the package and have added the common components, you need to save a copy of this package at the following locations based on the version of SQL Server you are using:

For SQL Server 2005
<<Installation drive>>:\Program Files (x86)\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

OR

<<Installation drive>>:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

For SQL Server 2008

<<Installation drive>>:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

OR
<<Installation drive>>:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

save a copy of this package at the following location based on version of SQL Server you are using

For SQL Server 2012
<<Installation drive>>:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

OR
<<Installation drive>>:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

You need to specify the drive location where Business Intelligence Development Studio (BIDS) or SQL Server Data Tools (SSDT) has been deployed. Please note, as BIDS or SSDT runs locally on client machine, you need to copy the template package to the above location on all the development machines you want it to use.  For this example we are naming the template package "SamplePackageTemplate.dtsx".

Simple Password Based Rijndael Encryption in .NET (C#)

 

This blog tells you how to encrypt and decrypt data

Link is here

http://blog.osamamirza.com/2011/10/simple-password-based-rijndael.html

Friday, June 21, 2013

How to add new package store under Stored Packages

 

In order to create a new folder called ‘My Packages’

1. Find the file called MsDtsSrvr.ini.xml in C:\Program Files\Microsoft SQL Server\100\DTS\Binn

2. Use a notepad and add the code in underline. Also make sure folder does exist. In this case c:\SSIS does exit.

 

3. Save and make sure you restart SSIS engine via Sql Server Configuration Manager

 

<?xml version="1.0" encoding="utf-8"?>
<DtsServiceConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <StopExecutingPackagesOnShutdown>true</StopExecutingPackagesOnShutdown>
  <TopLevelFolders>
    <Folder xsi:type="SqlServerFolder">
      <Name>MSDB</Name>
      <ServerName>.</ServerName>
    </Folder>


    <Folder xsi:type="FileSystemFolder">
      <Name>File System</Name>
      <StorePath>..\Packages</StorePath>
    </Folder>


    <Folder xsi:type="FileSystemFolder">
      <Name>My Packages</Name>
      <StorePath>C:\SSIS</StorePath>
    </Folder>


  </TopLevelFolders>
</DtsServiceConfiguration>

 

image

Monday, June 3, 2013

BIDS 2008 and TFS 2010

how to connect BIDS 2008 to TFS2010 for version control?


Check this blog:

http://microsoft-ssis.blogspot.ca/2012/04/ssis-2008-with-team-foundation-server.html

 

three steps1:

1. Install Team Explorer

2. install "Microsoft Visual Studio 2008 Service Pack 1" (even if it was already installed before step 1). You can download it here.

3. Thirdly install "Visual Studio Team System 2008 Service Pack 1 Forward Compatibility Update for Team Foundation Server 2010", which you can download here. This Service Pack let's you connect to TFS 2010.

Friday, May 10, 2013

Smalldatetime type issue

Recently, I’ve encountered an interesting situation.

createDateTime is Smalldatetime type

If using parameters , @datefrom and @end where they are DATETIME type, query is not running at all

After investigation, it appears that execution plan shows differently for query when using hardingcode and parameters.

In order to solve this issue, we have to use Query hint

 

When I cast DATETIME to SMALLDATETIME , it’s running …

select

COUNT(uf.uploadFileID),

--COUNT(ub.uploadBatchID),

ub.appContextID,

CONVERT(DATE,uf.createDateTime)

from dbo.UploadFile uf (nolock) JOIN dbo.UploadBatch ub (nolock)

on uf.uploadBatchID= ub.uploadBatchID

where uf.createDateTime >= @datefrom   ---->>>> small datetime type

and uf.createDateTime < @end

and uf.uploadFileStatusID = 0 -- success

and ub.uploadBatchStatusID = 0 -- success

and ub.appContextID in

(

2542704 --, -- previous android appcontextID notes like '%Partners 1993%'

--2167274 -- current android appContextID ac.notes like '%android%'

)

GROUP BY

ub.appContextID,

CONVERT(DATE,uf.createDateTime)

Friday, April 19, 2013

NCHAR(100) VS NVARCHAR(100)

 

 

this caused me a big headache during a SSIS package troubleshooting…

The problems is that the SSIS in my local is referring to Nvarchar(100) . When it’s deployed to the staging box, the column becoame nchar(100) which caused the ssis package to fail… gotta be consistent.

Thursday, April 18, 2013

Schema refresh in SSIS

 

Sometimes, SSIS package tends to remember older schema especially when connection manager is using expressions to be dynamic. Below is one example:

When you have modified schema on either source or target database, please make sure you’ve refresh .

 

For connection manager using expressions, you need to get rid of expressions and put a static first in order to refresh. Once it’s refreshed, put the expression back

 

image

 

Also, pay close attention to the initial value you put into variables. One example is like that: variable user::var_DBTimeZone comes from a table which contains dbTimeZone value. Currently its ‘vancouver’ which has 9 characters. But in the default value , I put ‘xxxxxxxx’ which has only 8 characters.  Therefore, when package is running, it tries to put data ‘Vancouver’ into this variables User::var_DBTimeZone which triggers the truncation error.

 

######################################################################################################

Schema info across all SSIS environment have to be consistent. Schema info refers to table name and schema name, not the server or database info as these will

be stored in the dynamic connection manager!!!!

#####################################################################################################

CHAR(39) refers to Apostrophe

 

in dynamic tsql, apostrophe was quite misleading, now, I tend to use Char(39) instead of '

Wednesday, February 20, 2013

SSIS Script task that FTP's files

use SSIS script task to perform FTP function

This is the article on the SSIS social forum

http://social.msdn.microsoft.com/Forums/en/sqlintegrationservices/thread/2b7cf6b5-d16d-44cd-954d-68e9ba98ee91

Friday, January 18, 2013

Top 10 SQL Server Integration Services Best Practices by Denny Lee

 

http://sqlcat.com/sqlCat/b/top10lists/archive/2008/10/01/top-10-sql-server-integration-services-best-practices.aspx

Wednesday, January 9, 2013

Issue about Could not open global shared memory to communicate with performance DLL

 

Recently, I am running a SSIS package on Win 7 and have the following error message:

 

Could not open global shared memory to communicate with performance DLL

Googled around and it turned out to be a bug and  and I need to install the CU package for my SSIS version which is R2 Sp1

 

http://support.microsoft.com/kb/2756574

Current version is

Microsoft SQL Server Integration Services Designer
Version 10.50.2500.0