Showing posts with label PowerShell. Show all posts
Showing posts with label PowerShell. Show all posts

Saturday, January 14, 2017

Powershell: Retrieving Qualtrics Survey Results Starter Script

Retrieving survey results from Qualtrics is less convenient than ever with the Qualtrics v3 API! 

Don't get me wrong, it's great that you can call a REST API to retrieve results from your survey. The inconvenient part is having to make three different API calls to get the results and ending up with a file that needs to be unzipped before you can work with it.

Thankfully, with PowerShell, this is a fairly straightforward task. Although there are many examples on how to do this using other programming languages, I was unable to find one using PowerShell, so I'm providing the script below as a starter script for your own purposes. You can easily add to this script if you need to export results in a different format (I only tested the below using "json" and "xml" as arguments) or wish to retrieve only a subset of responses or questions.

All of the information needed to create your API calls is nicely documented on the Qualtrics API site.


function Get-Qualtrics-Response-Export {
    param(
        [string] $organizationId,
        [string] $dataCenterId,
        [string] $apiToken,
        [string] $format,
        [string] $surveyId,
        [string] $outFile
        )

    $hostName = "{0}{1}qualtrics.com" -f "$organizationId.".TrimStart('.'), "$dataCenterId.".TrimStart('.')

    # ---------------------------------------------------------------------------- #
    #
    # XML - Create Response Export - Starts an export for a survey in XML format
    # https://api.qualtrics.com/docs/xml
    # https://api.qualtrics.com/docs/response-exports
    #
    # ---------------------------------------------------------------------------- #
    
    $url = "https://$hostName/API/v3/responseexports"
    
    $requestHeaders = @{}
    #$requestHeaders.Add("Accept", "*/*")
    $requestHeaders.Add("X-API-TOKEN", "$apiToken")
    $requestHeaders.Add("accept-encoding", "gzip, deflate")

    $requestData = @{
            format = "$format"
            surveyId = "$surveyId"
            includedQuestionIds = @()
            useLabels = $true
        }

    $body = (ConvertTo-Json $requestData)

    $response = Invoke-RestMethod -Method Post -Uri $url -ContentType "application/json" -Headers $requestHeaders -Body $body 
    $responseExportId = $response.result.id

    write-host "Response Export ID: $responseExportId"
    write-host
    
    # ---------------------------------------------------------------------------- #
    #
    # Get Response Export Progress - Retrieve the status of a response export
    # https://api.qualtrics.com/docs/get-response-export-progress
    #
    # ---------------------------------------------------------------------------- #

    $requestData = $null
    $body = (ConvertTo-Json $requestData)
    $percentComplete = 0
    $status = ""

    while($percentComplete -lt 100)
    {
        $response = Invoke-RestMethod -Method Get -Uri "$url/$responseExportId" -ContentType "application/json" -Headers $requestHeaders -Body $body 

        $status = $response.result.status

        if($status -eq "failed" -or $status -eq "cancelled")
        {
            # If you receive a status of cancelled or failed, please go back and regenerate a responseExportId 
            # for your desired export format - JSON, CSV, CSV 2013, XML, SPSS - and try this request again.
            write-host $response.result.info.reason
            write-host $response.result.info.nextStep
            break
        }

        $percentComplete = $response.result.percentComplete
        write-host "Response Export Progress: $percentComplete%"
    }

    # ---------------------------------------------------------------------------- #
    #
    # Get Response Export File - Retrieve the response export file after the export is complete
    # https://api.qualtrics.com/docs/get-response-export-file
    #
    # ---------------------------------------------------------------------------- #

    if($percentComplete -eq 100)
    {
        $response = Invoke-RestMethod -Method Get -Uri "$url/$responseExportId/file" -ContentType "application/json" -Headers $requestHeaders -Body $body -OutFile $outFile
        
        if(test-path $outFile)
        {
            $destination = split-path -path $outFile
            Expand-ZIP-File -zipFile $outFile -destination $destination
        }
    }
}

function Expand-ZIP-File($zipFile, $destination)
{
    $shell = new-object -com Shell.Application
    $zip = $Shell.NameSpace($zipFile)

    foreach($item in $zip.items())
    {
        $Shell.NameSpace($destination).CopyHere($item, 0x14)
    }
}

clear

<#
Your hostname should have one of the following formats:
    yourorganizationid.yourdatacenterid.qualtrics.com
    yourdatacenterid.qualtrics.com
    yourorganizationid.qualtrics.com
#>

$organizationId = "Your Organization ID (if exists)"
$dataCenterId = "Your Data Center ID (if exists)"
$apiToken = "Your API Token"
$format = "xml"
$surveyId = "Your Survey ID"
$outFile = "c:\ken\qualtrics\data\temp.zip"

#Get-Qualtrics-Response-Export -organizationId $organizationId -dataCenter $dataCenterId -apiToken $apiToken -format $format -surveyId $surveyId -outFile $outFile

Tuesday, August 12, 2014

Downloading and Extracting US Census Bureau TIGER/Line Shapefiles

The United States Census Bureau (census.gov) site is a great resource for free spatial data such as state, county, census tract, etc. boundary files. These files can be used as is or used as a building block for creating custom sales territories. They can be loaded into SQL Server for use in spatial queries or for displaying boundaries on a map in an SSRS report. They can also be used to build map tiles for use with the Bing Maps API. This post, however, is focused on downloading and extracting these files.

These TIGER (Topologically Integrated Geographic Encoding and Referencing)/Line Shapefiles are available in many vintages, including definitions from the 1990, 2000 and 2010 censuses, with annual updates through 2013 (2014 should be available soon). The Census Bureau site provides several ways to access and download these files, but the easiest way is to use their FTP site (ftp://ftp2.census.gov/).

You can use an FTP client or, as shown here, you can simply use Windows Explorer.

Start by opening Windows Explorer and pasting ftp://ftp2.census.gov/ directly into the address bar. 

Navigate to the geo/tiger/. From this folder you will see the different vintages of TIGER files available. To illustrate the next step, lets say we're interested in the 2013 Census Tract boundaries.

Navigate to the TIGER2013/TRACT folder. In the right hand window (i.e. file list), you'll see a list of compressed (.zip) files with names based on the State FIPS Code, which is a two character numeric code assigned to each state, the District of Columbia and outlying areas of the U.S. (Puerto Rico, Guam, etc.).

Select the ZIP files you're interested in. I'm only interested in the U.S. states and DC, so I'm selecting files tl_2013_01.zip (AL) through tl_2013_56.zip (WY).

Right-click one of the selected files and select Copy from the popup menu.

Navigate to a folder on your local network, right-click the folder and select Paste. This may take a few minutes. I'll wait.

Now that you have the files on your local network, we need to extract (i.e. unzip) them. Unfortunately Windows Explorer falls short when needing to unzip multiple files. If you already have WinZIP or WINRAR installed, you could use either of those to unzip multiple files. I don't, so I'm going to use the following PowerShell script that I found on Steve Schofield's Blog: 

function UnZipMe($zipfilename,$destination) 
{ 
    $shellApplication = new-object -com shell.application 
    $zipPackage = $shellApplication.NameSpace($zipfilename) 
    $destinationFolder = $shellApplication.NameSpace($destination) 

    # CopyHere vOptions Flag # 4 - Do not display a progress dialog box. 
    # 16 - Respond with "Yes to All" for any dialog box that is displayed. 

    $destinationFolder.CopyHere($zipPackage.Items(),20) 
} 

# replace the following with the folder that contains your zipped files
$a = gci -Path C:\Data\USCBCensusTracts2013 -Filter *.zip 

foreach($file in $a) 
{ 
    Write-Host "Processing - $file" 
    UnZipMe –zipfilename $file.FullName -destination $file.DirectoryName 
}

Once this script finishes running, you will have all of your shapefiles ready to work with.

Sunday, June 15, 2014

Creating Map Tiles (Part 3) - Generating Tiles from a TileMill Project Using MBUtil and PowerShell

This post is a continuation of Creating Map Tiles (Part 2) - Creating a Sales Territory Boundary Set Using TileMill, where I prepared my transparent boundary set in TileMill. In this post I use PowerShell to call TielMill and MBUtil to generate the map tiles.

As I mentioned in Part 1, for much of the process of creating map tiles I relied on an excellent series of posts by Pedro Sousa on his blog, Pedro's Tech Mumblings. In Part 2 of his series, he walks you through how to export your TileMill project to a MBTiles SQLite file and then use MBUtils (a command line MapBox tool) to export the map tiles. Below, I've taken a the same approach but created a PowerShell script to automate the process.

In order to perform this process manually or using the PowerShell script below, you need to install Python with SQLite and MBUtils. Pedro walks you through this, however I performed the following steps outlined in the accepted answer to Is there any way to use mbutil on windows? posted on StackExchange.
  • Download and install ActiveState Active Python 
  • Install MBUtil by downloading and unzipping to C:\mbutil-0.2.0.
  • From a command-line navigate to C:\mbutil-0.2.0 and run "setup.py install" to install mbutils
At this point the pre-requisites for the following PowerShell script exist. 

function CreateTiles() {
    param(
        [Parameter(mandatory=$true)] [string]$project,
        [Parameter(mandatory=$true)] [string]$inputPath,
        [Parameter(mandatory=$true)] [string]$outputPath
    )
    # delete .mbtiles, .export and .export-failed files
    remove-item  "$($inputPath)$($project).*"
    
    # delete directory
    $pngPath = $outputPath + $project
    if (test-path $pngPath){
       write-host "Deleting $($pngPath)... this may take awhile." 
       #remove-item $pngPath -recurse  # very slow
       $fso = New-Object -ComObject scripting.filesystemobject
       $fso.DeleteFolder($pngPath, $true)
    }
            
    # Create mbtiles file. Requires TileMill.
    # changed the following so that the executable's path doesn't 
    # contain spaces (i.e. using the call (&) operator is optional)
    cd "C:\Program Files (x86)\TileMill-v0.10.1\tilemill\"
    $mbtilesFile = "$($inputPath)$($project).mbtiles"
    & ".\node.exe" ".\index.js" "export" $project $mbtilesFile "--format=mbtiles"
    
    # Create PNGs from mbtile file. Requires:
    # Python with SQL Lite (install ActivePython);
    # MB Util (download mbutil zip from https://github.com/mapbox/mbutil/tree/v0.2.0).
    python "C:\mbutil-0.2.0\mb-util" --scheme=xyz $mbtilesFile $pngPath
}

CreateTiles "State_And_County_Boundaries" "C:\SHPs\" "C:\MapTiles\"

The first step in the script removes the existing MBTiles files, if they exist, which will block the creation of the new file. It also deletes the existing path of previously created tiles. This is necessary because I don't want to have any old tiles mixed with the new (remember not all tiles are generated within the bounds set in the TileMill project because my map has a transparent background). Depending on the number of tiles generated in a previous run, this can take awhile.

The second step is what you can do manually from TileMill by clicking the Export button in the main toolbar and selecting MBTiles. It calls TileMills index.js file and is documented here on the MapBox site.

The final step calls the MBUtils executable to produce the map tiles in a set of folders based on the XYZ tiling scheme.

The files now just have to be copied to a web server where they can accessed by the Bing Maps API. The following, modified from a function taken from another of Pedro's blog posts (taken in turn from a blog post by Alastair Aitchison), shows how the tiles can be called from the Bing Maps API using JavaScript:

var tileSource = new MM.TileSource({ 
    uriConstructor:  function getTilePath(tile) {
        // only created custom tiles for zoom 0-12
        if (tile.levelOfDetail <= 12) {
            var x = tile.x;
            var z = tile.levelOfDetail;
            var yMax = 1 << z;
            var y = yMax - tile.y - 1;

            return "Images/Tiles/BaseMap4/" + z + "/" + x + "/" + y + ".png";
        }
    }
});

var tileLayer = new MM.TileLayer({ mercator: tileSource, opacity: 1 });

// Push the tile layer to the map
map.entities.push(tileLayer);

One of the modifications that I made to the above prevents the call to retrieve tiles that were not generated for certain zoom levels. This will prevent the server from having to generate a multitude of 404 error response codes as you zoom below the level for which tiles were generated. Another modification that I made to my own code (not included above) was to add a version argument to the end of the URL which is retrieved from a web service call to the database when the page initially loads. I use this argument to force the fetching of new tiles when the tiles have been replaced on the server.


The Tile Layers Used with the Bing Maps API

Using the PowerShell script above, I have been able to generate three different, fairly static boundary sets, consisting of 3 to 5 layers each (resulting in around 100,000 tiles for each set). Along with the script from Part 1, Creating Map Tiles (Part 1) - Exporting MapInfo TAB Files to ESRI SHP Files Using PowerShell and MapInfo, I have been able to automate the creation of an additional two boundary map tile layers that have to be generated monthly, due to changes in the boundary definitions.


Wednesday, April 30, 2014

Creating Map Tiles (Part 1) - Exporting MapInfo TAB Files to ESRI SHP Files Using PowerShell and MapInfo

I recently had the requirement to display sales territory boundaries within a Bing Maps web application that I had written. The boundaries were actually a set of custom sales territories (e.g. region, district, markets) using counties as a building block. After some research, I determined the best way to incorporate these boundaries into my Bing Maps application was to generate a set of map tiles which would progressively show the boundaries (e.g. regions then districts then markets and finally counties) as you zoomed in.

For much of the process of creating map tiles, I relied on an excellent series of posts by Pedro Sousa on his blog, Pedro's Tech Mumblings. Starting with Part 2 of his series, he introduces you to TileMill and shows how it can be used to design a map.

One of the first steps after installing TileMill is to create a new project and add map layers to it, and one of the options for the map layers is to use ESRI Shape (.shp) files. I had a bit of a head start in the process as I had been using MapInfo and MapBasic to create MapInfo TAB files for many years for use in a desktop application that I had written back in the late 1990's. Because of my familiarity with MapInfo, I knew that it included a MapBasic tool called the Universal Translator which is a data conversion tool built on Safe Software's FME (Feature Manipulation Engine) technology. This could be used to translate my MapInfo TAB files into ESRI Shape files. However, I had several different boundary sets, consisting of several different boundaries (a dozen total), that I needed to create, and using an interactive tool would be tedious. I also had two boundaries that needed to be generated monthly, so automating the process was a must.



Because I could use MapBasic to write something to call the Universal Translator within MapInfo I knew I could automate the process. However, I got lucky when I unexpectedly stumbled upon the Universal Translator User Guide, which included information on how the MapInfo Universal Translator could be called from the command line. Armed with this information, I was able to quickly put together the following PowerShell script to convert all of my files.


function ExportTabToShp() {
    param(
        [Parameter(mandatory=$true)] [string]$TAB_PATH,
        [Parameter(mandatory=$true)] [string]$outputPath
    )

    $baseName = [System.IO.Path]::GetFileNameWithoutExtension($TAB_PATH)
    
    # Application name of the MapInfo Universal Translator
    $CMD = "C:\Program Files (x86)\MapInfo\Professional\UT\Imut.exe"
    
    # Command that generates a semantic control file
    $IMUT_COMMAND = "CFGenerate"
    
    # Input format is MapInfo .TAB
    $INPUT_FORMAT = "MAPINFO"
    
    # Output format is the ESRI Shapefile format
    $OUTPUT_FORMAT = "SHAPE"
    
    # Path and filename of the semantic control file
    $FME_PATH = $outputPath + $baseName + ".fme"
    
    #Path and filename of the logfile to be generated
    $LOG_FILENAME = $outputPath + "mut.log"

    # Generate a mapping file:
    write-host "Note the RemoteException that is thrown is simply a message stating that the 'Mapping File Generation' was successful:" -ForegroundColor Green
    & $CMD $IMUT_COMMAND $INPUT_FORMAT $OUTPUT_FORMAT $TAB_PATH $FME_PATH LOG_STANDARDOUT "YES" LOG_FILENAME $LOG_FILENAME LOG_APPEND "YES"

    # Run the mapping file:
    write-host "Note the RemoteException that is thrown is simply a message stating that the 'Translation' was successful:" -ForegroundColor Green
    & $CMD $FME_PATH --HOMOGENOUS_GEOMETRY "YES" --_SHAPE_FILE_TYPE "2d" --SourceDataset $TAB_PATH --DestDataset $outputPath --_EXTENSION "TAB" --_BASENAME $baseName --_FULLBASENAME $baseName
}

ExportTabToShp "C:\TABs\county.tab" "C:\SHPs\"

The first call in the ExportTabToShp function generates an FME file, which is a "mapping file" that controls the translation. The second call runs the translation using the generated FME file. Refer to the User Guide link above for more information required to expand this function to support other translations.

Note the use of the Call operator (i.e. &) in the above PowerShell script. This allows one to call an executable formatted as a string and is useful when there is a space in the path to the executable.

In the next part of this series, Creating Map Tiles (Part 2) - Creating a Sales Territory Boundary Set Using TileMill,I'll walk through a couple of the steps that I used in TileMill to create the sales territory boundary sets.

Tuesday, September 10, 2013

SSRS Beginner Tip: Generating PNGs from an SSRS Report (and PowerShell)

I recently had to generate a couple of dozen map images to be used by a client in their company website. The required map basically needed to be centered and zoomed on a sales territory boundary, highlighting it, displaying the trade areas within it and showing a Bing Maps background. SSRS wouldn't have necessarily been my first choice to create maps (MapInfo would have been), however I already had an SSRS report that contained this exact map. All I had to do was copy it and remove a couple of tables.

So setting up the SSRS report was pretty straight forward. In order to generate an image for each of the sales territories, I had to additionally create a Data-Driven Subscription, passing in the sales territory code into a report parameter. Again, pretty straight forward, until I realized that the only image export format available was a TIFF image file, which wouldn't work for the clients web page. They required a JPG or PNG.

The answer was to go ahead and generate the TIFFs using the subscription and then run this handy PowerShell script which would convert each image to a PNG:

#Load required assemblies and get object reference 
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms");

$path = "C:\MapImages"

Get-ChildItem $path *.tif |
    ForEach-Object { 
        $i = new-object System.Drawing.Bitmap($_.FullName);
        # Save with the image in the desired format
        $i.Save($("$($path)\$($_.BaseName).png"),"PNG"); 
    }

This script loops through each of the TIFFs located in the specified path, opening them and then saving them with the specified PNG format with the same base name as the TIFF. Most of this tiny script can be credited to the Hey, Scripting Guy! Blog post Hey, Scripting Guy! How Can I Use Windows PowerShell to Convert Graphics Files to Different File Formats?

Friday, February 22, 2013

The Broken “Export to Excel” Web Part Menu Item for PPS SSRS Web Parts and a Workaround

One of the side effects of using cumulative updates (CUs) is that functionality can break. I experienced this when applying the February 2012 CU to SharePoint 2010 SP1, which was required to fix some crucial functionality that was broken in SP1. The result, however, was that the "Export to Excel" web part menu item no longer worked for PerformancePoint Reporting Services web parts. The menu item is available, but instead of exporting, it opens the web part in the RSViewerPage.aspx page. Not real useful.

Upon some Googling, I found this forum entry indicating that the problem existed in the December 2011 CU and that it still exists with the December 2012 CU: http://social.technet.microsoft.com/Forums/en-US/ppsmonitoringandanalytics/thread/1d39d413-2756-450c-a40c-60c4dfe76fa1

As an additional note, this may be limited to an environment using SharePoint 2010 with SQL Server 2012 (this does work correctly with SharePoint 2010 SP1 with no CUs applied and SQL Server 2008 R2 SP1).

As the "Export to Excel" web part menu item was now rendered useless, I used PowerShell to loop through all of my dashboard pages, identifying the web parts that display PPS SSRS reports, and then hiding the "Export to Excel" menu item for those web parts so as not to confuse my dashboard users.

$webPart.IsAllowExportToExcel = "Hide" # Enabled, Disabled or Hide
$webPartManager.SaveChanges($webPart)

I left it at that for quite awhile, until in January I found time to revisit this problem and explore a workaround to allow the PPS SSRS web parts to be exported.

Something that I already knew worked correctly was the PPS SSRS web part’s toolbar’s exporting capability. So I could have just checked the "Show toolbar" checkbox in Dashboard Designer’s Report Settings for each of my SSRS web parts, but then the entire toolbar would display, taking up valuable real estate and providing unnecessary functionality for my dashboard web parts (e.g. page navigation, search, zooming, etc.), when I really only needed the exporting capability.

So instead I thought if I could dynamically show the toolbar, that would be an improvement over having it always being displayed. So again, with a little Googling, I found inspiration in the MSDN Library under an entry within the SQL Server 2012 Reporting Services Features and Tasks section called URL AccessParameter Reference. I could use query string arguments in the iframe element that displays the report to not only show the toolbar, but limit the toolbar functionality to just the exporting capability.

The next decision was how to incorporate this knowledge into my dashboard web parts. My first idea was to use jQuery to add a menu item, such as "Show Toolbar", to the web part’s menu, but I didn’t like how it would then take up real estate and cause my report to be partially obscured since I was setting the web part’s height. So instead I thought what if the toolbar only displayed when I opened the PPS SSRS web part in a new window using the web part’s "Open in New Window" menu item. Not perfect, but acceptable and relatively easy to accomplish. This would require editing the DynamicView.master and DynamicReportView.aspx files located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\PPSWebParts\ (I’ll leave it to you to determine how you would manage these two files and their deployment).
 
In the DynamicView.master file, I just had to add the PlaceHolderAdditionalPageHead server tag right before the closing HEAD tag (i.e. </head>):

<asp:ContentPlaceHolder id="PlaceHolderAdditionalPageHead" runat="server"/>

In the DynamicReportView.aspx file, I just had to add the following right before the "PlaceHolderMain" server tag:

<asp:Content ContentPlaceHolderId="PlaceHolderAdditionalPageHead" runat="server">
    <script src="../myScripts/jquery-1.7.1.min.js" type="text/javascript"></script>
    <!-- Shows Actions menu on toolbar (kb130521) -->
    <script type="text/javascript" language="javascript">
        function NotifyBrowserOfAsyncUpdate(elem) {
            $(elem).find('iframe[src*="%2Erdl"]').attr('src', function (i, val) {
                return val.replace('&rv:Toolbar=None&', '&rv:Toolbar=Full&rv:ToolBarItemsDisplayMode=128&')
            });
        }
        $(document).ready(function () {
            $('body').find('[text="Export to Excel"]').hide();
        });
    </script>
</asp:Content>

Like a couple of other blog posts that I have written, I’m using the NotifyBrowserOfAsyncUpdate function to notify the page when the asynchronous retrieval of the web part is complete. Then I use jQuery to find the iframe hosting the PPS SSRS web part and modify the "src" attribute to display the toolbar, but limit the menu items to just the export functionality. I additionally add a document ready function to hide the "Export to Excel" menu item that will appear in the web part's menu.

Now upon clicking a PPS SSRS web part menu's "Open in New Window" menu item, I get the following in a new window, showing the toolbar with just the Export menu option.

 
 

Thursday, January 31, 2013

Removing the Word "(Hidden) " from SharePoint Web Part Tooltips

When developing a dashboard in PerformancePoint 2010, one approach to managing the web parts that are displayed on the page is to use a PerformancePoint Filter to conditionally display web parts as a group. For example, I could have a filter that consists of the values "Market Share", "Dealers" and "Trend". By connecting this filter to web parts on the dashboard using the Display Conditions tab on the Configure Connection dialog, I can display certain web parts when "Market Share" is selected, others when "Dealers" is selected, etc.

The problem arises when, after a page refresh, you click on your filter to display a different group of web parts. The tooltip will now have the word "(Hidden) " prefacing the name of the web part in the tooltip for the web part menu, the web part icon and the web part description (see below).


Clearly the web part is not hidden. For whatever reason, this is indicating the initial state of the web part after a full page postback. Helpful? Not Really. Annoying? Definitely.

With a little jQuery, we can easily remove that word from all web parts in the document ready function.


$(document).ready(function () {
    // Remove "(Hidden) " from a title, img alt and td title tags
    $('td[title^="(Hidden) "]').each(function (index) {
        $(this).attr('title', $(this).attr('title').replace('(Hidden) ', ''));
    });
    $('img[alt^="(Hidden) "]').each(function (index) {
        $(this).attr('alt', $(this).attr('alt').replace('(Hidden) ', ''));
    });
    $('a[title^="(Hidden) "]').each(function (index) {
        $(this).attr('title', $(this).attr('title').replace('(Hidden) ', ''));
    });
});

There is certainly room for targeting the selector better here, but I did want to show that there are several tags that need to be addressed. Either way  the result is the same: a less confusing dashboard for your users.

Sunday, September 23, 2012

Importing a SharePoint 2010 .CMP file and the “String was not recognized as a valid DateTime” Error

There are plenty of excellent posts out there on how to do a SharePoint 2010 granular backup and restore operation on a document library using PowerShell. A good overview is provided in this blog post on MSDN: SharePoint 2010 Granular Backup-Restore Part 1. I use this process to deploy a set of custom dashboards that I’ve created.

I've put together a couple of PowerShell functions to export (i.e. backup) and import (i.e. restore) a .CMP file (again, plenty of great examples out there), and they work great. However, I ran into a problem when importing one of my document libraries, which lead me to discover an error, similar to the following, in the corresponding log file:

[9/19/2012 1:21:50 PM] [ListItem] [MyDashboard.aspx] Progress: Importing
[9/19/2012 1:21:50 PM] [ListItem] [MyDashboard.aspx] Verbose: List URL: /aSiteCollection/aSite/DashboardPages
[9/19/2012 1:21:50 PM] [ListItem] [MyDashboard.aspx] Error: String was not recognized as a valid DateTime.
[9/19/2012 1:21:50 PM] [ListItem] [MyDashboard.aspx] Debug: at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)
   at System.String.System.IConvertible.ToDateTime(IFormatProvider provider)
   at ...

After a little Googling, I discovered this forum post on TechNet which identified the problem and a workaround: String was not recognized as a valid DateTime on import of site
.

The problem turns out to be in the Manifest.xml file located within the .CMP. The actual line in the .XML causing the problem contains an erroneous year (i.e. 60354):

<Property Name="vti_syncwith_smartbidev\:80/ymus" Type="Time" Value="29 May 60354 05:36:10 -0000" Access="ReadWrite"/>

And now for the purpose of this post: the following PowerShell script basically renames the .CMP file to a .CAB, extracts the files into a folder, searches and replaces the erroneous date in the Manifest.xml and imports it using import-spweb with the NoFileCompression switch.
 
function ImportCAB {
    param(
        [Parameter(mandatory=$true)] [string]$webUrl,
        [Parameter(mandatory=$true)] [string]$fileName
    )    

    try
    {
        $cabFile = $fileName.Replace(".cmp", ".cab")

        if(!(test-path $fileName)) 
        {
            write-host "$fileName does not exist. Exiting..."
            return
        }

        write-host "Copying $fileName to CAB file"
        copy-item $fileName $cabFile
        write-host "Copied $fileName to CAB file"

        $cabFolder = "$(split-path $fileName -Parent)\$((Get-Item -path $fileName).basename)"
        
        if(test-path $cabFolder) { remove-item $cabFolder -recurse }
        
        New-Item -ItemType directory -Path $cabFolder
    
        # Creating CAB Files with Windows PowerShell
        # http://lab.technet.microsoft.com/en-us/magazine/dd547834#id0070049
        $comObject = "Shell.Application" 
        write-host "Creating $comObject" 
        $shell = New-Object -Comobject $comObject 
        if(!$?) { $(Throw "unable to create $comObject object")} 
        write-host "Creating CAB object for $cabFile"
        $sourceCab = $shell.Namespace($cabFile).items()
        write-host "Creating destination folder object for $cabFolder" 
        $DestinationFolder = $shell.Namespace($cabFolder)
        write-host "Expanding $cabFile to $cabFolder" 
        $DestinationFolder.CopyHere($sourceCab)
        
        # Search Manifest.xml for the following and replace "60354" with a valid year
        $findText = "Value=`"29 May 60354 05:36:10 -0000`""
        $replaceText = "Value=`"29 May 2012 05:36:10 -0000`""
        
        (Get-Content "$($cabFolder)\Manifest.xml") | 
            Foreach-Object {$_ -replace $findText, $replaceText} | 
            Set-Content "$($cabFolder)\Manifest.xml" -encoding UTF8

        import-spweb -identity $webUrl -path $cabFolder -force -UpdateVersions 2 -nofilecompression #-whatif
    }
    catch [Exception]
    {
        write-host $_.Exception.ToString() -ForegroundColor Green
    }
}

This works rather well, but does not address the real issue of how to fix the source so that the export produces a valid .CMP file in the first place. In theory, I can import this back into my development machine and fix the issue (try at your own risk).

Wednesday, April 11, 2012

SharePoint Designer 2010 and PerformancePoint Web Part Connections (Part 3)

Part 1, Part 2, Part 3

In the two previous parts to this post, I provided a function to connect two PerformancePoint Services (PPS) web parts and a function to connect a SharePoint Filter to a PPS web part. These functions work great, but only if you know the arguments to pass. These arguments can be GUIDs, for example, in the case of connecting a PPS web part to a Scorecard's row or setting a PPS web part's Display Condition based on a PPS Filter.

In this post, I'll solve that problem for you by providing a function that can create all of the function calls for you for a web part page that has existing connections on it. The purpose of this function is to record the connections on the page so that when they are lost (e.g. by saving a web part page using SharePoint Designer), the script can recreate them in seconds.

The Create-Connection-Script function below loops through all of the connections on a web part page, identifying the type connections and creates a script full of function calls to the Create-BIDataProvider-To-TransformableBIDataProvider-Connection and Create-IFilterValues-To-ITransformableFilterValues-Connection functions. If you create the connection script prior to editing and saving a web part page in SharePoint Designer, you'll be able to run the newly created script afterwards and your PPS web parts will be reconnected.

An example of how to call the Create-Connection-Script function for a web part page with exisitng connections:

# Script that contains functions
. M:\ps1\connections\ConnectWebParts.ps1

clear-host

try
{
    $scriptFolder = "M:\ps1\connections"

    if (!(Test-Path -path $scriptFolder))
    {
        New-Item $scriptFolder -type directory
    }

    $date = Get-Date -format "yyMMddHHmmss"

    $web = Get-SPWeb "http://servername/sitecollectionname/webname"
    $wpm = $web.GetLimitedWebPartManager($web.Url + "DashboardPages/Demo.aspx", [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
    $scriptPath = Join-Path $scriptFolder -childpath "$($date)DemoConnections.ps1"    
    Create-Connection-Script $wpm $scriptPath
}
catch [Exception]
{
    write-host $_.Exception.ToString() -ForegroundColor Red
}
finally
{
    $wpm.Dispose() 
    $web.Dispose()
}   
 
The function in it's entirety (place this in a .PS1 file along with the functions provided in Parts 1 and 2):

function Create-Connection-Script {
    param(
        [Parameter(mandatory=$true)] $wpm,
        [Parameter(mandatory=$true)] [string]$scriptPath
    )

    try
    {
        $m_scriptPath = $scriptPath
        clear-script
        
        $wpConnections = $wpm.SPWebPartConnections
        write-script "# This script was generated by $($MyInvocation.MyCommand)" -Color Green
        write-script "# Script that contains functions"
        write-script ". M:\ps1\connections\ConnectWebParts.ps1"
        write-script "# Script for $($wpConnections.Count) connection(s) on web part page ($($wpm.ServerRelativeUrl))." -Color Green
        write-script ""
        write-script "try"
        write-script "{"
        write-script "    `$web = `Get-SPWeb `"$($web.Url)`""
        write-script "    `$wpm = `$web.GetLimitedWebPartManager(`"$($web.Url.Replace($web.ServerRelativeUrl, $wpm.ServerRelativeUrl))`", [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)"
        write-script ""
        
        write-host $web.Url.Replace($web.ServerRelativeUrl, $wpm.ServerRelativeUrl)
        
        foreach ($wpc in $wpConnections)
        {
            write-script "    # $($wpc.Consumer.Title) To $($wpc.Provider.Title)" -Color Green

            if($wpc.ConsumerConnectionPointID -eq "BIDataProvider" -and $wpc.ProviderConnectionPointID -eq "TransformableBIDataProvider")
            {
                write-script "    `$params = @{"
                write-script "        wpm = `$wpm"
                write-script "        wpConsumerTitle =`"$($wpc.Consumer.Title)`""
                write-script "        wpProviderTitle =`"$($wpc.Provider.Title)`""
                
                foreach($record in $wpc.Transformer.ConfigurationState.ProviderConsumerTransformations.Records)
                {
                    write-script "        consumerParameterName = `"$($record.ConsumerParameterName)`""
                    write-script "        providerParameterDisplayName = `"$($record.ProviderParameterDisplayName)`""
                    write-script "        providerParameterName = `"$($record.ProviderParameterName)`""
                    write-script "        providerFormula = `"$($record.ProviderFormula)`""
                }

                $cvr = $wpc.Transformer.ConfigurationState.ConditionalVisibilityRecord         

                if($cvr.IsDefined) 
                {
                    write-script "        cvrIsDefined = `$True"
                    write-script "        cvrProviderParameterDisplayName = `"$($cvr.ProviderParameterDisplayName)`""
                    write-script "        cvrProviderParameterName = `"$($cvr.ProviderParameterName)`""

                    $cvrVisibilitySelections = ""
                    foreach($vs in $cvr.VisibilitySelections)
                    {
                        if($cvrVisibilitySelections.length -ne 0)
                        {
                            $cvrVisibilitySelections += ","
                        }
                        $cvrVisibilitySelections += "`"$vs`""
                    }
                    write-script "        cvrVisibilitySelections = @($($cvrVisibilitySelections))"
                    write-script "        cvrIsDefaultVisibility = $([System.Convert]::ToInt32($cvr.IsDefaultVisibility))"
                }                
                write-script "        reconnect = `$False"
                write-script "    }"
                write-script "    Create-BIDataProvider-To-TransformableBIDataProvider-Connection @params"
            }
            elseif($wpc.ConsumerConnectionPointID -eq "IFilterValues" -and $wpc.ProviderConnectionPointID -eq "ITransformableFilterValues")
            {
                write-script "    `$params = @{"
                write-script "        wpm = `$wpm"
                write-script "        wpConsumerTitle =`"$($wpc.Consumer.Title)`""
                write-script "        wpProviderTitle =`"$($wpc.Provider.Title)`""
                write-script "        mappedConsumerParameterName =`"$($wpc.Transformer.MappedConsumerParameterName)`""
                write-script "        reconnect = `$False"
                write-script "    }"
                write-script "    Create-IFilterValues-To-ITransformableFilterValues-Connection @params"
            }
            else
            {
                write-host "    # Web parts did not have a supported connection" -ForegroundColor Green
            }
            write-script ""
        }
        
        write-script "}"
        write-script "catch [Exception]"
        write-script "{"
        write-script "    `write-host `$_.Exception.ToString() -ForegroundColor Red"
        write-script "}"
        write-script "finally"
        write-script "{"
        write-script "    `$wpm.Dispose()"
        write-script "    `$web.Dispose()"
        write-script "}"
    }
    catch [Exception]
    {
        write-host $_.Exception.ToString() -ForegroundColor Red
    }    
}

function Write-Script {
    param(
        [Parameter(mandatory=$false)] [string]$text,
        [System.ConsoleColor] $color="Gray"
    )
    
    #$text = "$(Get-Date -format g): $text"
    
    write-host $text -ForegroundColor $color
    
    out-file $m_scriptPath -Append -InputObject $text
}
 
Additionally, you'll note the write-script function at the bottom, which I use to write to a file and to the screen.

Hopefully, you can use these functions the way that I have been using them for the past several months, or you can use them to create your own functions to make custom dashboard pages easier to maintain.

Thursday, April 5, 2012

SharePoint Designer 2010 and PerformancePoint Web Part Connections (Part 2)

Part 1, Part 2, Part 3

In Part 1 of this post, I provided a function that can reconnect PerformancePoint Services (PPS) web parts as an alternative to having to reconnect the web parts via the browser. This can come in handy when you've lost all of your PPS connections after having saved your web part page in SharePoint Designer.

In this post, I'm providing a function, Create-IFilterValues-To-ITransformableFilterValues-Connection, that can connect a SharePoint Filter to PPS web parts. I had to create this function to connect Query String (URL) Filters (QSUF) to my PPS web parts. Now this wasn't totally necessary, as these connections are not destroyed by saving your page in SharePoint Designer, but I have used it in various scenarios (e.g. deployment, creating new pages from existing) and it is referenced in the function that I'll provide in Part 3.

An example of connecting a QSUF to a PPS SSRS report:

try
{
    $web = Get-SPWeb "http://servername/sitecollectionname/webname"
    $wpm = $web.GetLimitedWebPartManager("http://servername/sitecollectionname/webname/DashboardPages/demo.aspx", [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)

    # Report: Share by Market Scatter Plot To Area Filter
    $params = @{
        wpm = $wpm
        wpConsumerTitle ="Report: Share by Market Scatter Plot"
        wpProviderTitle ="Area Filter"
        mappedConsumerParameterName ="SqlReportViewUniqueParameterIdSI4"
        reconnect = $False
    }
    Create-IFilterValues-To-ITransformableFilterValues-Connection @params
}
catch [Exception]
{
    write-host $_.Exception.ToString() -ForegroundColor Red
}
finally
{
    $wpm.Dispose()
    $web.Dispose()
}
Here is the function in its entirety:

function Create-IFilterValues-To-ITransformableFilterValues-Connection {
    param(
        [Parameter(mandatory=$true)] $wpm,
        [Parameter(mandatory=$true)] [string]$wpConsumerTitle,
        [Parameter(mandatory=$true)] [string]$wpProviderTitle,
        [Parameter(mandatory=$true)] [string]$mappedConsumerParameterName,
        [bool]$reconnect=$false
    )
    
     try
    {
        $foundConnection = $false
        
        $wpConnections = $wpm.SPWebPartConnections

        # find connection
        foreach ($wpc in $wpConnections)
        {
            if ($wpc.Consumer.Title -eq $wpConsumerTitle -and $wpc.Provider.Title -eq $wpProviderTitle)
            {
                if ($reconnect -ne $true) {
                    $foundConnection = $true
                }
                else {
                    $wpm.SPDisconnectWebParts($wpc)
                }
                break
            }
        }

        if($foundConnection -eq $false)
        {
            # get consumer and provider web parts
            $wpConsumer = $wpm.WebParts | Where {$_.Title -eq "$wpConsumerTitle"}
            $wpProvider = $wpm.WebParts | Where {$_.Title -eq "$wpProviderTitle"}
 
            # get consumer and provider connection points
             $consumerConnectionPoint = $wpm.GetConsumerConnectionPoints($wpConsumer)["IFilterValues"]
             $providerConnectionPoint = $wpm.GetProviderConnectionPoints($wpProvider)["ITransformableFilterValues"]

             if($consumerConnectionPoint -ne $null -and $providerConnectionPoint -ne $null)
             {
                 $transformer = New-Object Microsoft.SharePoint.WebPartPages.TransformableFilterValuesToFilterValuesTransformer
                 $transformer.MappedConsumerParameterName = $mappedConsumerParameterName

                 # connect the web parts
                 $wpConnection = $wpm.SPConnectWebParts($wpProvider, $providerConnectionPoint, $wpConsumer, $consumerConnectionPoint, $transformer) 

                 write-host "Successfully connected: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor Green
             }
             else
             {
                 write-host "Web parts did not have supported ITransformableFilterValues-To-IFilterValues connection types: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor Green
             }
        }
        else
        {
             write-host "Connection already exists: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor DarkCyan
        }
    }
    catch [Exception]
    {
        write-host $_.Exception.ToString() -ForegroundColor Green
    }
    finally
    {
    }    
}
In Part 3 of this post, I'll provide a function that uses the functions from Part 1 and this part to create a connection script for you when you provide it a web part page full of connections.

Thursday, March 29, 2012

SharePoint Designer 2010 and PerformancePoint Web Part Connections (Part 1)

Part 1, Part 2, Part 3

When using PerformancePoint Services (PPS) web parts in a web part page to create custom dashboards, you’ll typically set up connections between the (PPS, OOTB and custom) web parts using your web browser. Once your connections are established, you’ll realize that you don’t want to run through the process more than once, especially if you have tens of connections on the page. For a project that I am currently working on, I have several dashboards consisting of around 80 connections each (thankfully, I didn't have to create all of the connections at once).

What I quickly discovered (as many have) is that if you have to use SharePoint Designer to customize the web part page after the PPS connections have been established, you'll run into a very painful problem. It turns out that upon saving a web part page with PPS connections in SharePoint Designer, all of the PPS connections are lost.

My solution, after a lot of Googling and trial and error, has been to create PowerShell functions that connect PPS web parts (Part 1), connect PPS web parts to SharePoint filters (Part 2), and create connection scripts by having PowerShell examine the connections on a web part page (Part 3).

The Create-BIDataProvider-To-TransformableBIDataProvider-Connection function, provided below, will handle the following connections:
  • Scorecard to PPS Filter
  • Analytic Report (i.e. Chart or Grid) to PPS Filter
  • Analytic Report to Scorecard
  • PPS SSRS Report to PPS Filter
  • PPS SSRS Report to Scorecard
It will also handle Connection Formulas and Display Conditions. There's certainly room for improvement, but it handles everything that I need it to do regarding PPS web part connections.

An example of calling the function to connect a Scorecard to a PPS Filter:

try
{   
    $web = Get-SPWeb "http://servername/sitecollectionname/webname"
    $wpm = $web.GetLimitedWebPartManager("http://servername/sitecollectionname/webname/DashboardPages/demo.aspx", [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)

    $params = @{
        wpm = $wpm
        wpConsumerTitle ="Scorecard"
        wpProviderTitle ="Area Filter"
        consumerParameterName = "EndPoint_Page"
        providerParameterDisplayName = "PerformancePoint Values"
        providerParameterName = "FilterValues"
        providerFormula = ""
        reconnect = $False
    }
    Create-BIDataProvider-To-TransformableBIDataProvider-Connection @params
}
catch [Exception]
{
    write-host $_.Exception.ToString() -ForegroundColor Red
}
finally
{
    $wpm.Dispose()
    $web.Dispose()
}

I won't bore you with a bunch of examples, because in Part 3, I'll provide a script that will take your web part page and create all of the function calls for you.

Below is the function is in entirety:

# cvr parameters: ConditionalVisibilityRecord Properties
function Create-BIDataProvider-To-TransformableBIDataProvider-Connection {
    param(
        [Parameter(mandatory=$true)] $wpm,
        [Parameter(mandatory=$true)] [string]$wpConsumerTitle,
        [Parameter(mandatory=$true)] [string]$wpProviderTitle,
        [string]$consumerParameterName="",
        [string]$providerParameterDisplayName="",
        [string]$providerParameterName="",
        [string]$providerFormula="",
        [bool]$cvrIsDefined=$false,
        [string]$cvrProviderParameterDisplayName="",
        [string]$cvrProviderParameterName="",
        [string[]]$cvrVisibilitySelections = @(),
        [bool]$cvrIsDefaultVisibility=$false,
        [bool]$reconnect=$false
    )
    
     try
    {
        $foundConnection = $false
        
        $wpConnections = $wpm.SPWebPartConnections

        # find connection based on consumer and provider web part titles 
        foreach ($wpc in $wpConnections)
        {
            if ($wpc.Consumer.Title -eq $wpConsumerTitle -and $wpc.Provider.Title -eq $wpProviderTitle)
            {
                if ($reconnect -ne $true) {
                    $foundConnection = $true
                }
                else {
                    $wpm.SPDisconnectWebParts($wpc)
                }
                break
            }
        }

        # if found, attempt the connection process
        if($foundConnection -eq $false)
        {
            # get the consumer and provider web parts
            $wpConsumer = $wpm.WebParts | Where {$_.Title -eq "$wpConsumerTitle"}
            $wpProvider = $wpm.WebParts | Where {$_.Title -eq "$wpProviderTitle"}
           
            # get the consumer and provider connection points
            $consumerConnectionPoint = $wpm.GetConsumerConnectionPoints($wpConsumer)["BIDataProvider"] 
            $providerConnectionPoint = $wpm.GetProviderConnectionPoints($wpProvider)["TransformableBIDataProvider"]
            
            [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.PerformancePoint") 

            if($consumerConnectionPoint -ne $null -and $providerConnectionPoint -ne $null)
            {
                $pcTransformations = New-Object Microsoft.PerformancePoint.Scorecards.ProviderConsumerTransformations

                # note: the consumerParameterName may be empty if a Display Condition only
                if($consumerParameterName.length -ne 0)
                {
                    # establish the connection settings
                    $tpcRecord = New-Object Microsoft.PerformancePoint.Scorecards.TransformProviderConsumerRecord
                    $guid = [guid]::NewGuid()
                    $tpcRecord.MappingId = $guid.ToString()
                    $tpcRecord.ConsumerParameterName = $consumerParameterName
                    $tpcRecord.ProviderParameterDisplayName = $providerParameterDisplayName
                    $tpcRecord.ProviderParameterName = $providerParameterName
                    $tpcRecord.TypeFullName = "System.String"
                    $tpcRecord.ProviderFormula = $providerFormula
                    $tpcRecord.ValuesColumnName = "MemberUniqueName"
                    $tpcRecord.DisplayColumnName = "DisplayValue"
                    $tpcRecord.EncodeAsSet = $false
                    $pcTransformations.Records.Add($tpcRecord)
                }
                
                # conditional visibility
                $tcvRecord = New-Object Microsoft.PerformancePoint.Scorecards.TransformConditionalVisibilityRecord
                $tcvRecord.IsDefined = $cvrIsDefined
                $tcvRecord.ProviderParameterDisplayName = $cvrProviderParameterDisplayName
                $tcvRecord.ProviderParameterName = $cvrProviderParameterName
                foreach($vs in $cvrVisibilitySelections)
                {
                    $tcvRecord.VisibilitySelections.Add($vs)
                }
                $tcvRecord.IsDefaultVisibility = $cvrIsDefaultVisibility

                $tcRecord = New-Object Microsoft.PerformancePoint.Scorecards.TransformerConfigurationRecord($pcTransformations, $tcvRecord)
                
                $transformer = New-Object Microsoft.PerformancePoint.Scorecards.WebControls.TransformableBIDataProviderTransformer
                $guid = [guid]::NewGuid()
                $transformer.ID = $guid.ToString()
                $transformer.ConfigurationState = $tcRecord

                # finally create the connection
                $wpConnection = $wpm.SPConnectWebParts($wpProvider, $providerConnectionPoint, $wpConsumer, $consumerConnectionPoint, $transformer) 
                #$wpConnections.Add($wpConnection)

                 write-host "Successfully connected: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor Green
             }
             else
             {
                 write-host "Web parts did not have supported BIDataProvider-To-TransformableBIDataProvider connection types: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor Green
             }
        }
        else
        {
             write-host "Connection already exists: $wpConsumerTitle, $wpProviderTitle" -ForegroundColor DarkCyan
        }
    }
    catch [Exception]
    {
        write-host $_.Exception.ToString() -ForegroundColor Green
    }
    finally
    {
    }    
}
 
In the part of this post, I'll provide a function that connects SharePoint Filters to PPS web parts.