How to export dynamically calculated values to a file
In my previous post I talked about MDX Insert which allows you to store dynamically calculated values in the cube, but what if you want to export it into a file? There are a lot of different reasons to do so, but the most common one would be moving this data into a different application. Back in 2016 Gary Crisci explained how it can be done using MAXL and MDX.
The solution proposed in the post did the job, but I wasn't too happy about it for two reasons.
First would be the query governor - by default Essbase kills queries that return too much data. It is done to avoid runaway queries that exhaust all server memory. Second - the file is created on the client machine which means we need to send the data over and do all the formatting work on the client. In addition if the file needs to be shared with multiple consumers, user would have to upload it to some shared location. So we implemented MDX Export.
I'll use the same Zigzag application from previous posts to illustrate how it works. Remember the "Closing Inventory" and "Opening Inventory"?
“Closing Inventory” = "Opening Inventory" - "Shipped" + "Received";
“Opening Inventory” =
IF(("Inventory on hand" != #Missing) OR (@ISMBR(Jan)))
"Inventory on hand";
ELSE
@PRIOR("Closing Inventory");
ENDIF;
To query these two members we can use following MDX query:
SELECTTo run the query open the Analyze screen for the Zigzag.demo cube. Click "Execute MDX" button and enter the query in to MDX dialog.
{[Opening Inventory], Shipped, Received, [Closing Inventory]} ON COLUMNS,
NON EMPTY CrossJoin(Time.Levels(0).members,
CrossJoin(Dim1.Levels(0).members,
CrossJoin(Dim2.Levels(0).members, Dim3.Levels(0).members))) ON ROWS
FROM Zigzag.demo
Click Execute button and see the query results
Now let's modify the query to save the results into a file. Simply add following on top of our original query
EXPORT INTO FILE "demo" OVERWRITE USING COLUMNDELIMITER ","
and click Execute again. This time the result grid will look like below
It means that the query was successfully executed. Navigate to the data base inspector and download the demo.txt file
The file should have the same content as the query result above
Time,Dim1,Dim2,Dim3,Opening Inventory,Shipped,Received,Closing InventoryFor more details about MDX export see Official documentation
Jan,1-11,2-1,3-1,100,10,15,105
Feb,1-11,2-1,3-1,105,23,12,94
Mar,1-11,2-1,3-1,94,48,12,58
Apr,1-11,2-1,3-1,58,1,10,67
May,1-11,2-1,3-1,67,60,90,97
Jun,1-11,2-1,3-1,97,75,1,23
Jul,1-11,2-1,3-1,23,22,100,101
Aug,1-11,2-1,3-1,101,93,100,108
Sep,1-11,2-1,3-1,108,10,50,148
Oct,1-11,2-1,3-1,148,132,13,29
Nov,1-11,2-1,3-1,29,20,100,109
Dec,1-11,2-1,3-1,109,#Missing,#Missing,109
Hi Victor.Nice Post
ReplyDeleteTrying to uderstnad the two issues that you have mentioned
First would be the query governor - by default Essbase kills queries that return too much data. It is done to avoid runaway queries that exhaust all server memory.
Do you have any hard-stop of how much memory would be consumed and by what extent it can fail? I believe it is not the pending cache limit which is 32MB by default? I had queries that extract a million records and didn't fail. However, as it was a calculated export, the query time was longer
Second - the file is created on the client machine which means we need to send the data over and do all the formatting work on the client. In addition if the file needs to be shared with multiple consumers, user would have to upload it to some shared location.
I think the catch here is if you are running the MaxL on client machine. If you are executing it on the server, you can specify the spool on the server itself and still get away with a decent time
One thing I didn't like about the MaxL formatting and MDX query is that if you setup the SET options in a way to write only the data to spool to the file, you never know if something fails as that won't be returned to the console and you have to set the message level to all for debugging purpose.
I hope MDX Export solves this issue by logging the output to the spool file in MaxL and the MDX output to the file specificed in the MDX Export syntax provided the MDX Export is run via a MaxL
Also, is it possible to run multiple MDX queries parallel and export to multiple files?
Hi Victor,
ReplyDeleteAs I have mentioned at KScope this year, the fastest way to export large sub-cube that I have found is to actually export the entire cube to server side file with MaxL "export database level0 data to data_file" (important - no report script, default export format), parse that and load it in an SQL table (in my case with a C# SSIS Script task just because I needed to load in SQL Server), and then index the table and load back from it to Essbase with load rules + MaxL and SSIS for automation. For some reason the data export in native file format is blazing fast, in my case took 1 minute. Comparing that with more than 1 hour using MDX reports ran in parallel (groups of 8-16 processes) and all the trouble to do the scripting to slice the sub-cube in manageable MDX reports (in my case had to use looping by lev0 in 2 dimensions to generate the dynamic MDX reports, queue those, pull them for the parallel runs batches with the proper process sync on the queue, parse and aggregate the results from the MaxL spool ... not a trivial job).
I assume it makes sense that exporting database in default format is fast, as it probably bypasses everything and just dumps to file the internal storage of the ASO cube.
... and I should re-iterate here my complaint about undocumented features in Essbase: nowhere I could find the format of the database export documented. Had to do trial and error to figure it out, load the cube back and check the numbers match. I think it would add so much to the value of Essbase as a tool if it would be more "open book". If there are features that are under development and subject to change, or not supported ("don't open SRs, use at own risk"), it can be mentioned in documentation page as disclaimer, same as Microsoft does all over MSDN, but still have them documented and save me and many others countless hours of poking a black box ...