Configuring the BizTalk application

This is the final step of this process. In order to properly test the solution we have been developing in BizTalk Server, we need to configure the application that was created in the publication.

For this, we will access the BizTalk Server Administration console to create and configure the following artifacts:

• One receive port and location to process the input text files;
• One send port to save the processed files in XML format..

Open the BizTalk Server Administration console under “Start –> Programs –> Microsoft BizTalk Server 2010”. On the left tree menu, maximize “BizTalk Server Administration –> BizTalk Group (…) –> Applications”, and look for the application called “TransfFlatFilesEmXML Demo” and maximize also this application.

• Press the right button on top of “Receive Port” menu and select “New –> One-Way Receive Port …” option;
• A new window appears allowing us to define the port properties:
  • In the “General” tab, set the name of the receive port: “ReceiveTXT”.

• In the “Receive Locations” tab, select “New” and set the name to your receive location: “ReceiveTXT_to_XML”; in the property “Type” select the option FILE; and in the property “Receive pipeline” select from the drop down box the pipeline we created earlier: “ReceivePipelineCustomPessoas”;

• On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set “Receive Folder” property, with the folder you created earlier “<solution>\PORTS\IN”. On property “File Mask” put the following value: “*. txt”, setting that will be processed only files with the extension. “txt”. Finally select the “Ok” button.

• To finish the process of creating the receive port press “Ok”.

The above steps mentioned, were needed to create one Receive Port and its Receive Location. Now we will create a Send Port to save messages on the file system, after they are processed by BizTalk:

• Press the right button on top of “Send Ports” menu and select “New –> Static One-way Send Port…”
• A new window appears allowing us to define the port properties:
  • In the “General” tab, set the name of the send port: ” SaveXMLFile “
  • In the property “Type” select the option FILE;
  • In the property “Send Pipeline” set the pipeline to “XMLTransmit”. This is a native pipeline that is used for processing XML messages.

• On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set the “Destination Folder” property, with the folder you created earlier “<solution>\PORTS\OUT”. On property “File Mask” put the following value: “%MessageID%.xml”, this is special tag (BizTalk Macro) that will set the name of each file written on the file system with the unique identifier of the message. Finally select the “Ok” button to return to previous window.

• In order for the Send Port subscribes to all files, we need to set a filter on the tab “Filters” with the following expression: “BTS.ReceivePortName == ReceiveTXT”.
  • Note: This setting will force each message in the MessageBox which has originates from the Receive Port: ReceiveTXT, be routed and processed by the Send Port we are finished creating.

• To finish the process of creating the send port press “Ok”.

Finally, we need only to start our application. To do that press right button on top of the application name “TransfFlatFilesEmXML Demo” and select “Start …”.

Testing the solution

To test the BizTalk application we only need to copy a text file to the directory that is configured on the receive location: “<solution>\PORTS\IN”. The result should appear in the form of an XML document in the folder configured in the send port: “<solution>\PORTS\OUT”.

Note: The file disappears automatically from the input folder, so you should avoid doing drag & drop (move) with the test files at the risk of not being able to use it again.

Conclusion

As presented in this article, BizTalk Server helps us to solve many of the existing problems concerning message transformation: semantic and syntax Transformations, only with “out of the box” features within the product, in few steps and most of the time without developing any code.

The fact of all the definitions of the rules of parsing (transformation rules) are embedded in the form of annotations in XML Schema (XSD), such as the delimiter symbols, or the element size in a positional file, simplifies the reuse of all these schemes in different parts of the process. At any point, the document can be translated back into flat-file because the definition is declarative and symmetric, in contrast to what might happen with a “normal” programming language like C#, where we would have to create two components to translate text to XML and vice versa.

The model used by BizTalk schemas, , centralizes all the message definitions in a single artifact, simplifying the maintenance and evolution of the specification and therefore simplifying future reuse and dependencies management. When we set up this type of project, is exactly this kind of guarantees than keeps the systems operating properly for many years, and at the same time, they make life easier to those who one day have to correct and/or develop any of these processes

Related links

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)

Source code

The solution of this example, as well all the code, is available on MSDN Code Gallery:
BizTalk Server – Transform text files (Flat Files) into XML (64.5 KB)
MSDN Code Gallery

Following one of my last speaking engagement in the Porto.Data Community Meeting about Flat Files: How to process Flat Files documents (TXT, CSV …) in BizTalk Server where I address some of the following topics:

• How to process Flat Files documents (TXT, CSV …) in BizTalk Server.
• What types of flat files are supported?
• How is the process of transforming text files (also called Flat Files) into XML documents (Syntax Transformations)
• Where does it happen and which components are needed
• Or how can I perform a flat file validation?
• And so on

I decided to return writing about this topic in my blog, a topic that I already wrote in the past describing and explaining almost all of the topics mention above:

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)
• BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

But this time transforming my previous session in a sequence of blog posts where I will address and explain how to solve some of the typical or common flat files structures and scenarios.

The simplest ones:

• Text Files Delimited by symbols
  • For example:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4400
José;Silva;1972;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

• Where we have: Name,Surname,Birthyear,Address,ZipCode
• Positional Text Files
  • For example:

0120200300 01 PT        Sandro      Pereira         Rua Central, 133      Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        José        Silva           Crestuna              Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        Rui         Barbosa         Porto                 Porto       Porto              4400
0120200300 01 PT        Miguel      Silva                                 Porto       Porto              4415 Crestuma

• Where we have:

GroupId|PrimaryKey|SecondaryKey|ControlFlag|PartyNo|Remark|Country|Name|Surname|Street|City|District|ZipCode

0120|20|03|0|0 |01 |PT |Sandro |Pereira |Rua Central, 133 |Crestuma |Vila Nova de Gaia |4415 Crestuma

And evolve to more challenger scenarios like:

• Handling Record Tag Identifiers, were we will treated each Tag Identifier differently
  • For example:

PO1999-10-20
USERSandro Pereira,1978-04-04, Crestuma
USERCelso Pereira,1978-04-04, Crestuma
ITEM872-AA|Super Bock|Green
ITEM871-AA|Super Bock|Black

• Where we have:
  • Header
    • Date
  • User
    • Name|Birthyear|Address
  • Item
    • Id|Product|Specification
• Dealing with Headers and Trailers
  • For example: (see the original post here: Walkthrough: Flat File Disassembly Using a Header and Trailer)

East Coast Facility
ERROR102|0|High|Sprocket query fails.|1999-05-31T13:20:00.000-05:00
ERROR16502|2|Low|Time threshold exceeded.|1999-05-31T13:20:00.000-05:00
8675309

• Where we have:
  • Location
  • ERROR
    • ID|Type|Priority|Description|DateTime
  • BatchID
• Or simple How can I remove a simple header from a Text File
  • For example

P_ABB,NUMBER,SHIP_ID,CREATE_DATE,SHIP_NO,SHIP_DATE,SHIP_DEST,PO_NO,LOT_TYPE,LOT_NO,SHIP_QTY,SHIP_PRD_NO,OBS_LOT
PA1,IN1900011,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PD806-D0,AGT;AKT;LOOP
PA1,IN1900012,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PM806-D0.0J,AGT;AKT;FLAT

• Where the structure of the message is described in the header (first line) – This scenario for example is very common in CSV files.

Until we reach and address advance topic like:

• Debatching Flat Files messages
• Perform Deep Validation in flat files processes
• Create generic Receive and Send Pipelines and how to configures this pipelines through BizTalk Administration Console
• And so on

And hopefully teaching you something new (trick or tip) about Flat Files or at least trying to demystify some issues related with Flat Files.

Most of this things will be solved by explaining you how to create, for each scenario, flat file schema from flat file document instances using the BizTalk Flat File Schema Wizard.

Others how can we tuning some properties afterwards so that we can have the desired output.

Let’s the fun begin.

This post is intended to consolidate some basic principles about Flat File Schema Wizard

First of all, BizTalk Flat File Schema Wizard is a template inside Visual Studio that allows you to converting Flat File into XML message, which BizTalk can understand, in a graphical a simple manner. That basically supports two types of text files: Positional text files and/or delimited by symbols. See more:

• BizTalk Server: Transform text files (Flat Files) into XML–Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)

The following walkthrough shows you how to create a flat file schema from a semicolons separate flat file document that stores people information using the BizTalk Flat File Schema Wizard.

File sample:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4400
José;Silva;1972;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

You can see that we have, in this example, four records with five fields specifying Name, Surname, Birth year, Address and Zip code of each person.

Now I will walk you through the required steps at design time:

• using the Flat File Schema Wizard to generate the Flat File Schema;
• and the pipeline editor to create a custom pipeline;

To enable the runtime to support the processing of the Flat File described earlier (People)

We begin then by launching Visual Studio and create or open a BizTalk project:

• “File –> New –> Project”, on BizTalk Projects, select the option “Empty BizTalk Server Project”;
• Insert the project name, physical location on the disk and the name of the solution.

Using Flat File Schema Wizard to generate a schema based on a file delimited by Symbols

To create the schema which will recognize the ‘people’ text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”

• On “Installed Templates” menu in the window “Add New Item”, select the option “Schema Files”, and then select the option “Flat File Schema Wizard”, then provide the name you want to give the scheme in this example: “FF_DSYMBOL_PEOPLE.xsd”

By selecting this option, we will be guided automatically by the tool “BizTalk Flat File Schema Wizard” that will help us to create a Flat File Schema and define its data structure (records, elements, attributes …) based on the text file specified.

• Click “Next” to continue.

• In the window “Flat File Schema Information” we will have to:
  • Select an instance of the text file that will serve as the model of the structure that we want to transform;
  • Although it is not necessary, it is good practice to rename the Record name “Root”.
    • In this case we will rename it to “People”
  • And finally, assign a “Target namespace” to the scheme and define the encoding of the input file
  • Click “Next” to continue.

Note: The wizard will load the text file so that we can begin to split it and map it into the desired structure. In this step we need to define how the records or rows are differentiated. The structure of the example is: Name;Surname;Birthdate;Address;Zip Code{CR}{LF}

• Since each record “Person” that we want to create is defined in a single line, in the “Select Document Data” we can select only the data portion of the document that will set the record, i.e. the whole first line. Click “Next” to continue.

Note: Alternatively, let’s call it option 2, we could select all the data of the file, however with this approach we need to treat differently the data a few steps ahead in the “Child Elements” window.

• In the window “Select Record Format” we will define whether we are dealing with a Flat File Delimited by symbols or is positional, in our case we will select the “By delimiter symbol”, because each record is delimited by a return Carriage Return/Line Feed ({CR}{LF}). Click “Next” to continue.

• In the window “Delimited Record” we will provide the record delimiter, in this case as we want to define the structure of person, i.e. each row is a person, our limiter is {CR}{LF} (Carriage Return/Line Feed). The “Child delimiter” combo box include by default the following options: {CR}, {LF}, {TAB}, {SPACE}, {0x1A}, {}, {.}, {;} and {CR}{LF}, however you can provide your own symbol.
  • Click “Next” to continue.

• In the window “Child Elements” we will define what kind of element we want to assign to the registry. As we are defining the Person structure and the file contains multiple people, we have to select the “Element Type” as “Repeating record” If we do not perform this step, we will not have the ability to split the record into multiple elements / attributes individual.
  • Note: If you do not define the element as “Repeating record” (or “Record”), you will not have the ability to break down your record into individual elements or attributes.
  • Click “Next” to continue.

Note: earlier I told that we could select all the data inside the flat file and that we would need to treat the information differently in this step – option 2. So if you select this option, it will present all the lines in the document in this window, because we want to have an “Array” of Persons, in this step we need to select the “Element Type” of the first element as “Repeating record” and the rest of the set as “Ignore” (You set this to Ignore because you have already specified Repeating Record for the Person element)

In this point, I have defined how the element Record looks like but I have not broken down the various elements that make up the record itself. We have just map that each line of the text file corresponds to a record Person.

• In the “Schema View”, click “Next” to continue processing the message. Click “Next” to continue.

In this phase the wizard restarts the whole process described above, but if you noticed, the wizard no longer selects all information contained in the text file, but only what was selected to define the record Person.

• What we will do now is split the information of record “Person” in different elements, for that, we will select only the information required leaving out the Carriage Return/Line Feed. Click “Next” to continue.

• Once again our structure is delimited by symbols (;), then we will select the option “By delimiter symbol” and then c“Next” to continue.
• As we can analyze all the elements are separated by the semicolon (;) that is our delimiter, then in the window “Delimited Record” we must change the value of the “Child delimiter” option to “;”.

• In this window “Child Elements”, we will define the different elements/attributes of the structure of the record person. This operation is very similar to any XSD, where we can define the different names and data types. Adjust the values ​​according to the table below:

Note: You should rename the parent node, each record, element or attribute in the BizTalk Flat File Schema Wizard, mainly in this step by modify the “Element Name” field to your desired name, however if you forgot to do that or if you make a mistake defining one of them, you can always rectify it in the Schema Editor. The same applies to the Data Type.

• Click “Next” to continue.
• Finally the wizard will show the equivalent XML structure that your text file document will have. Once you select the option “Finish”, the scheme will be available for you to use in your BizTalk solution.

After we finalize the creation of the Flat File Schema which will contain the transformation rules of the text file, we can easily test our transformation, without having to get out of our development tool (Visual Studio) and without having to deploy our solution.

If we select the Flat File Schema that we just created and access to its properties, we can see that by default all properties are preconfigured so that we can perform tests to our message transformation:

• The input instance file is configured with the file which was used to create the schema;
• And with the correct input and output formats
  • Validate Instance Input Type: Native
  • Generate Instance Output Type: XML

To test, you simply right-click in the schema that we just created and select the option “Validate Instance”:

This option will use the configured file to execute and validate all transformation rules defined in the Flat File Schema and present the final result or occurring errors in the “Output” window

Did you know that: Delimited fields in flat files have a limit of 50000000 characters

Creating the Pipeline that will be responsible for processing and transforming the text file

As I explained in detail in one of my previous posts: BizTalk Server: Transform text files (Flat Files) into XML–Introduction (Part 1), the Syntax Transformations occur in the Disassembling and Assembling stages within the receive and send pipelines, having this Flat File schema will not do us much good until we create the pipelines to provide the translation between Flat Files and XML

To create the Receive Pipeline that will be responsible for processing and transforming the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Pipeline Files”, and then select the option “Receive Pipeline”, then provide the name you want to give the pipeline in this example: “FF_People_ReceivePipeline.btp”

• By selecting the option “Add”, the pipeline editor (BizTalk Pipeline Designer) will appear and that will let you view and add components to all the steps associated with receive pipeline: Decode, Disassemble, Validate and ResolveParty.

In this case, the pipeline that we will create, will be responsible for receiving a text file through a receive location and convert it to XML. For that, we will use the “Flat File Disassembler” component which is available in the Visual Studio Toolbox window (Ctrl+W, X).

• Drag it into the Pipeline in step “Disassemble”.

• Finally, select the component “Flat file disassembler”, go to its properties and on the “Document Schema” option: select the schema created earlier, in case the “FF_DSYMBOL_PEOPLE”.

Note: If you want to create a send pipeline in order to transform an XML document into Flat File, we would follow the same steps, the difference is that we would have to drag the component “Flat File Assembler” on stage “Assemble”.

Once you have deployed your solution, you will want to ensure that you select the custom pipeline we just created in the Receive Locations that you expect to receive these flat files.

Take away

Now the big questions here is: Do you think the schema is entirely correct? Does it handles with all files accordingly to the expected results?

Or do we need to tune some of the properties of your schema… stay tuned to the next post.

Some references:

• Flat File Messages with Delimited Records
• How to Configure the Flat File Disassembler Pipeline Component
• BizTalk Flat File Schema Wizard

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction

You can download all the code used here from:

BizTalk Server: Dealing with Files Delimited by Symbols (71,9 KB)
Microsoft | Code Gallery

We ended last post asking: Do you think the schema is entirely correct? Does it handles with all files accordingly to the expected results? Or do we need to tune some of the properties of your schema…

When we start working with Flat File schemas for BizTalk, we may think that just using the Flat File wizard is going to be enough. Once the schema is created, we just need to create a custom pipeline and add the schema to the Document Schemas of a Flat File disassembler and it will be good… well let me tell you that most probably is not enough and the reality may be very different.

A common mistake that we sometimes do is to rely on a single sample message to generate/validate our Flat File Schemas. And we ended up getting possibly several different errors in production due to unforeseen cases that end up not being supported in our current schema version.

So it is real important to get all the information, as possible, from the different teams, for example:

• Are there any optional elements?
• If so, will they be delete from the message? Or will they be present without any content (blank/null)
• Does the message will always end with a blank line? i.e., does the last line contain a Carriage Return/Line Feed (CR+LF)
• What are the Data type of the elements?

And many, many more….

The following walkthrough shows you some of the common scenarios that you can expect and how you can tune some of the properties of our Flat File Schemas to deal with them.

Cosmetic operations

While generating the Flat File Schema with the Flat File Schema Wizard you should always provide/set:

• The correct or proper namespace for the schema
• Rename the parent node, each record, element or attribute
• And set the correct data type of each element or attribute

However if you forgot to do that or if you make a mistake defining one of them, you can always easily rectify it in the Schema Editor.

I call it cosmetic operations because they don’t infer “any changes” in the translation rules generated by the wizard, they only changed the way the elements are presented in the message, if you change the name of the element or attributes. Or they will have effect in the validation of the message, if we change the data types.

You can accomplished that by:

• Open the schema in BizTalk Editor.
• Change the Schema Target Namespace
  • In the BizTalk Editor dialog, right-click the <Schema> node in the tree view and click Properties. The Properties dialog appears.
  • In the Properties dialog, the Target Namespace displays under the General category for the <Schema> node.
  • In the Target Namespace field, type the name of your new target namespace and click OK.
• Change the Field Attribute or Field Element Name
  • Select the Field Attribute or Field Element node and then right-click and select the “Rename” option
  • Type the desired Field Attribute or Field Element name and press “Enter”
• Change the Field Attribute or Field Element Data Type
  • Select the Field Attribute or Field Element node and then right-click and select the “Properties” option
  • In Properties window, set the “Data Type” property to the desired type

Validating Flat Files Schemas through the Visual Studio with different scenarios

Let us forget the runtime for now, and focus firstly on the development phase where we can test and validate our schemas directly from Visual Studio as explained, once again, in the previous post.

Dealing with Files ending with a newline or that doesn’t ends with a newline

Initially we had the following example:

Sandro;Pereira;1978;Crestuma;4415{CR}{LF}
Vânia;Braziela;1981;Vila Nova de Gaia;4400{CR}{LF}
José;Silva;1972;Crestuma;4415{CR}{LF}
Rui;Barbosa;1975;Lever;4415{CR}{LF}

Which was the base to generate our Flat File schema: “FF_DSYMBOL_PEOPLE.xsd”.

If you paid attention, while we were generating our Flat File Schema, that in the “Select Document Data” window of the BizTalk Flat File Schema Wizard, you can notice that the last line contains a Carriage Return/Line Feed (CR+LF), which means that if you open the file for example in Notepad, you will notice that the file ends with a blank line.

Note: You can check the file “FF_DSYMBOL_PEOPLE.txt” that is available with the source code.

So let us try with a simple change by removing the Carriage Return/Line Feed so that the files does not end with a newline:

Sandro;Pereira;1978;Crestuma;4415{CR}{LF}
Vânia;Braziela;1981;Vila Nova de Gaia;4400{CR}{LF}
José;Silva;1972;Crestuma;4415{CR}{LF}
Rui;Barbosa;1975;Lever;4415

Check the file: “FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt”

If we now try this file instance against our Flat File Schema by:

• In Solution Explorer, right-click the schema that you want to validate, and then click Properties.
• In Properties window for the schema, for Input Instance Filename enter the name and path for the “FF_DSYMBOL_PEOPLE
  • Check more about Schema File Properties here.
• WithoutEnterLastLine.txt” message instance, or browse to the file, select it, and then click OK.
• Right-click the schema that you want to validate, and then click Validate Schema

You will notice that we will get the following error:

Invoking component…
<path>\DemoFile\FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt: error BEC2004: Unexpected data found while looking for:
‘\r\n’
The current definition being parsed is Person. The stream offset where the error occured is 133. The line number where the error occured is 4. The column where the error occured is 23.
<path>\FF_DSYMBOL_PEOPLE.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE.xsd, file: <file:///path\FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt>.
Component invocation succeeded.

If, on the other hand, we had originally generated the Flat File Schema using the message instance without the Carriage Return/Line Feed at the end.

And if we know try it with an instance containing a blank line at the end of the message we would receive a similar but different error:

Invoking component…
<path>\FF_DSYMBOL_PEOPLE.txt: error BEC2004: Unexpected end of stream while looking for:
‘;’
The current definition being parsed is People. The stream offset where the error occured is 139. The line number where the error occured is 5. The column where the error occured is 0.
<path>\FF_DSYMBOL_PEOPLE_Advance.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE_Advance.xsd, file: <file:///path\FF_DSYMBOL_PEOPLE.txt>.
Component invocation succeeded.

I’ve seen situations/solutions where there were created pipeline components to address and solve these type of problems, which isn’t the best and the proper solution!

In both cases the solution for solving these problems is the same: ensure that the “Allow Message Breakup At Infix Root” property on the Schemas node is set to “true” by:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, right-click in the Schema node and select the Properties option
• In Properties window for the schema, set the “Allow Message Breakup At Infix Root” property to “true”

After making this change, if we try to validate the two previous messages instances, we will notice that now everything will work perfectly.

So if you notice in the properties of our Schema:

• The “Child Order” property of the People Root Node is set as: “Postfix”
• And the “Child Order” property of the “Person” record is set as: “Infix”

What does it means? Child Order must always have a value. The possible values are:

• Prefix: Specifies that the delimiter appears before the data that it is delimiting.
• Postfix: Specifies that the delimiter appears after the data that it is delimiting.
• Infix: Specifies that the delimiter appears between different items of data, resulting in one less delimiter than when Prefix or Postfix is chosen.
• Default Child Order: Sets the child_order attribute to “default”, specifying that the child order specified using the Default Child Order property should be used.
• And Conditional Default: Removes the corresponding annotation in the XSD representation of the schema, if any, resulting in the following behavior:
  • If the Record node has a tag identifier, then the child order will be prefix.
  • Otherwise, the child order will be infix.

There are two scenarios related to delimited flat files for which special considerations apply when setting the Child Order property.

• The first such scenario concerns situations in which the flat file document has a header, a body, and optionally, a trailer.
• The second scenario is that this property must be set according to what the runtime components expect for the nodes

In our instance message, the data/delimiter sequence in the conceptual loop would merely be one or more person records. Of course, only in the case where there are more than one person records would there be a delimiter to separate them. In that case, the number of delimiters is one less than the sets of things being delimited, and the delimiters are located between the delimited items in a structure known as Infix.

• Number delimiter = Number Records – 1

Which is not the case in all our messages instances. Our first instance have the same number of “valid records” and delimiters.

By setting “Allow Message Breakup At Infix Root” property to “true”, this tells the parser whether to allow message breakup, and will ensure that the Infix stream is processed properly by the FF Disassembler.

Dealing with Optional Elements

Using Suppress Empty Nodes property

So let’s imagine a different scenario, this time we will deal with optional elements. In our first scenario while discussing with our time we realize that the element “Birthyear” was appearing empty in several messages, and it should be considered as an optional element:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

By default, we need to have a complete instance of the message, for us developers be able to create all the transformation rules, which means that by default, when we finish creating the Flat File Schema through Flat File Schema Wizard tool, none of the records, elements or attributes are optional.

All of these changes/optimizations/configurations, they have to be made after the Flat File Schema been created.

You can always modify an existing Flat File schema that was created by the wizard (name of the elements, data type and so on). So to accomplish that we need to:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, expand People root node, expand Person record and right-click in the Birthyear element and select the Properties option
• In Properties window, set the “Min Occurs” property to “0”

If we now try to validate the previous message instance (“FF_DSYMBOL_PEOPLE WithOptional.txt”), we ended up receiving another error:

Invoking component…
Validation generated XML output <file:///<path>\FF_DSYMBOL_PEOPLE_Advance_output.xml>.
<path>\FF_DSYMBOL_PEOPLE_Advance_output.xml: error BEC2004: The ‘Birthyear’ element is invalid – The value ” is invalid according to its datatype ‘http://www.w3.org/2001/XMLSchema:int&#8217; – The string ” is not a valid Int32 value.
<path>\FF_DSYMBOL_PEOPLE_Advance.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE_Advance.xsd, file: file:///path\FF_DSYMBOL_PEOPLE WithOptional.txt.
Component invocation succeeded.

And of course the reason is obvious, an empty string is not a valid integer.

Making this element nillable doesn’t make any difference, because in both situations the parser will continue to add an empty element:

<Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear></Birthyear>
    <Address>Crestuma</Address>
    <ZipCode>4415</ZipCode>
  </Person>

To solve that situation there are a few additional flat file node properties that we can use. You can see the full list here:

• Schema Node Properties

And in our case we are looking for the “Suppress Empty Nodes” property (or suppress_empty_nodes). By setting this property to “True” it will remove all the empty nodes from the XML stream.

• This can be used to eliminate fields that are empty after being parsed, but the XSD type doesn’t allow empty values.

To accomplish that we need to:

• In BizTalk Editor, right-click in the Schema node and select the Properties option
• In Properties window for the schema, set the “Suppress Empty Nodes” property to “true”

After making this change, if we try to validate the previous message instance, we will notice that now everything will work perfectly.

Note: When using Delimited files, the optional element in the beginning or in the middle of the each line (record node) need always to be present in the text with empty values (;;) otherwise the parsing rules will fail or will perform an incorrect translation of the message.

If the “Birthyear” is set as string and we receive this type of messages

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

The message will be translated incorrectly and there is nothing you can do about it, the end result would look like this:

  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>Crestuma</Birthyear>
    <Address>
      4415
      Rui
    </Address>
    <ZipCode>Barbosa;1975;Lever;4415</ZipCode>
  </Person>

Dealing with Optional Elements (last element with the data type string)

Continue in the same scenario/message type, after a further series of tests, we noticed another unforeseen situation. Once again several messages were being executed incorrected because some of the element were not being parsed the element “ZipCode” – note that this element is a string

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;1972;Crestuma
Rui;Barbosa;1975;Lever;4415

Note: Contrary to what happens with elements at the beginning and end of the message, if the last element is an optional element, then it is not required to appear in the text (see line 3 – it has one element less)

We don’t receive any error message in BizTalk Server runtime but we notice that we messages are being integrated with incorrect data. After further analysis, let’s take the sample above, we notice that the message are being translated incorrectly and the last node is being added to the “ZipCode” element and not in a different Person node as supposed

  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>1972</Birthyear>
    <Address>
      Crestuma
      Rui
    </Address>
    <ZipCode>Barbosa;1975;Lever;4415</ZipCode>
  </Person>

To solve this problem we once again need set the element, in this case “ZipCode” as optional:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, expand People root node, expand Person record and right-click in the ZipCode element and select the Properties option
• In Properties window, set the “Min Occurs” property to “0”

<People xmlns="http://DelimitedBySymbols.FF_DSYMBOL_PEOPLE_Advance">
  <Person xmlns="">
    <Name>Sandro</Name>
    <Surname>Pereira</Surname>
    <Birthyear>1978</Birthyear>
    <Address>Crestuma</Address>
    <ZipCode>4415</ZipCode>
  </Person>
  <Person xmlns="">
    <Name>Vânia</Name>
    <Surname>Braziela</Surname>
    <Birthyear>1981</Birthyear>
    <Address>Vila Nova de Gaia</Address>
    <ZipCode>4300</ZipCode>
  </Person>
  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>1972</Birthyear>
    <Address>Crestuma</Address>
  </Person>
  <Person xmlns="">
    <Name>Rui</Name>
    <Surname>Barbosa</Surname>
    <Birthyear>1975</Birthyear>
    <Address>Lever</Address>
    <ZipCode>4415</ZipCode>
  </Person>
</People>

There is a Flat File Schema property call “allow_early_termination” (Early Terminate Optional Fields) that is a Boolean property. The use of this property is this situation is incorrect and does produce any changes in outcome. We will address and explain this property at another time.

Some references:

• Document Validation in the Flat File Disassembler Pipeline Component
• Schema Node Properties
• Additional Flat File Properties

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction
• BizTalk Server: Teach me something new about Flat Files (or not) – Files Delimited by Symbols

You can download all the code used here from:

BizTalk Server: Dealing with Files Delimited by Symbols (96 KB)
Microsoft | Code Gallery

In the second post of this series we follow a simple walkthrough explaining the basic principles to create a flat file schema from a file delimited by symbols. Now it’s time to do the same thing, but this time translating an inbound positional flat-file.

A positional flat file it’s normally a file that has fields that have a fixed maximum length and records that have a common end-of-record terminator, for example:

Sandro      Pereira         {CR}{LF}
José        Silva           {CR}{LF}

Where the first fixed 12 characters describe the Name field and the following 16 characters describe the Surname field. White space padding the end of each field and the Carriage Return/Line Feed characters is the common end-of-record terminator.

The following walkthrough shows you how to create a flat file schema from a positional flat-file document that stores employee information using the BizTalk Flat File Schema Wizard.

File sample:

0120200300 01 PT        Sandro      Pereira         Rua Central, 133      Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        José        Silva           Crestuna              Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        Rui         Barbosa         Porto                 Porto       Porto              4400 Crestuma
0120200300 01 PT        Miguel      Silva                                 Porto       Porto              4415 Crestuma

Once again I will walk you through the required steps at design time:

• using the Flat File Schema Wizard to generate the Flat File Schema;
• and the pipeline editor to create a custom pipeline;

To enable the runtime to support the processing of the Flat File described earlier (Employees)

We begin then by launching Visual Studio and create or open a BizTalk project:

• “File –> New –> Project”, on BizTalk Projects, select the option “Empty BizTalk Server Project”;
• Insert the project name, physical location on the disk and the name of the solution.

Using Flat File Schema Wizard to generate a schema based on a positional flat-file

To create the schema which will recognize the ‘Employees’ text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”
• On “Installed Templates” menu in the window “Add New Item”, select the option “Schema Files”, and then select the option “Flat File Schema Wizard”, then provide the name you want to give the scheme in this example: “FF_Positional_Employees.xsd”

Once again, has we explain previous, by selecting this option, we will be guided automatically by the tool “BizTalk Flat File Schema Wizard” that will help us to create a Flat File Schema and define its data structure (records, elements, attributes …) based on the text file specified.

• Select “Next” to continue.

• In the window “Flat File Schema Information” we will have to:
  • Select an instance of the text file that will serve as the model of the structure that we want to transform;
  • Although it is not necessary, it is good practice to rename the Record name “Root”.
    • In this case we will rename it to “Employees”
  • And finally, assign a “Target namespace” to the scheme and define the encoding of the input file

Note: The wizard will load the text file so that we can begin to split it and map it into the desired structure. In this step we need to define how the records or rows are differentiated.

• Since each record “Employee” that we want to create is defined in a single line, in the “Select Document Data” we can select only the data portion of the document that will set the record, i.e. the whole first line.

Note: Alternatively, let’s call it option 2, we could select all the data of the file, however with this approach we need to treat differently the data a few steps ahead in the “Child Elements” window.

Note: if you notice that some characters are not being parsed correctly as the picture bellow shows, then you probably need to go back to the “Flat File Schema Information” windows and set a different code page, for example “Western-European (1252)” instead of “UTF-8 (65001)”.

• In the window “Select Record Format”, even though we are dealing with a positional flat-file, at this point we are trying to define the structure of each “Employee” record witch is delimited by a return Carriage Return/Line Feed ({CR}{LF}). So in this case we will have to select the “By delimiter symbol” option.

• In the window “Delimited Record” we will provide the record delimiter, once again, we want to define the structure of Employee, i.e. each row is an Employee node, our limiter is this sample will be the {CR}{LF} (Carriage Return/Line Feed).

• In the window “Child Elements” we will define what kind of element we want to assign to the registry. As we are defining the Employee structure and the file contains multiple employees, we have to select the “Element Type” as “Repeating record” If we do not perform this step, we will not have the ability to split the record into multiple elements / attributes individual.
  • Note: If you do not define the element as “Repeating record” (or “Record”), you will not have the ability to break down your record into individual elements or attributes.

Note: earlier I told that we could select all the data inside the flat file and that we would need to treat the information differently in this step – option 2. So if you select this option, it will present all the lines in the document in this window, because we want to have an “Array” of Employees, in this step we need to select the “Element Type” of the first element as “Repeating record” and the rest of the set as “Ignore” (You set this to Ignore because you have already specified Repeating Record for the Employee node)

In this point, we have defined the translation rules for the repeating record “Employee” but we have not broken down the various elements that compose the record itself. We have just map that each line of the text file corresponds to a record Employee.

• In the “Schema View” select “Next” to continue processing the message

At this point everything is exactly the same as we did in the Delimited by Symbols example. From here forward is that the changes will emerge.

• In this phase the wizard restarts the whole process described above, but if you noticed, the wizard no longer selects all information contained in the text file, but only what was selected to define the record Employee. What we will do now is split the information of record “Employee” in different elements, for that, we will select only the information required leaving out the Carriage Return/Line Feed. On the “Select Document Data” screen click “Next”.

• On the “Select Record Format” screen, because now the content of the “Employee” record needs to be parsed according to predefined length (positional), we need to select the option “By relative positions” and then click “Next”

• The wizard provides a visual tool for showing and calculating the distance between the fields. On the “Positional Record” page, Use the left mouse button (or the right) to click a position marker in the position selection box to set the new position marker line. The position marker is represented as a solid line in will represent the rule in which a specific name field begins.
  • Position Marker selection box: Click an area on the page to represent the beginning position of the element. Click again to remove the position marker.
  • By default, a position marker line exists at the beginning of the record at position zero.
  • Click the following position markers to represent the rest of the data fields:

• In the “Child Elements” page, we will define the different elements/attributes of the structure of the record person. This operation is very similar to any XSD, where we can define the different names and data types. Adjust the values to:
  • Have the same name of the table above in the “Element Name”
  • Leave the default “Element Type” and “Date Type” – Field element and String

Note: Once again, you should rename the parent node, each record, element or attribute in the BizTalk Flat File Schema Wizard, mainly in this step by modify the “Element Name” field to your desired name, however if you forgot to do that or if you make a mistake defining one of them, you can always rectify it in the Schema Editor. The same applies to the Data Type.

• Select “Next” to continue.
• Finally the wizard will show the equivalent XML structure that your text file document will have. Once you select the option “Finish”, the scheme will be available for you to use in your BizTalk solution.

After we finalize the creation of the Flat File Schema which will contain the translation/transformation rules of the text file, we can easily test our parsing rules, without having to get out of our development tool (Visual Studio) and without having to deploy our solution.

If we select the Flat File Schema that we just created and access to its properties, we can see that by default all properties are preconfigured so that we can perform tests to our message transformation:

• The input instance file is configured with the file which was used to create the schema;
• And with the correct input and output formats
  • Validate Instance Input Type: Native
  • Generate Instance Output Type: XML

To test, you simply right-click in the schema that we just created and select the option “Validate Instance”. This option will use the configured file to execute and validate all transformation rules defined in the Flat File Schema and present the final result or occurring errors in the “Output” window

Did you know that Fields in positional records have a limit of 50000000 characters?

Creating the Pipeline that will be responsible for processing and transforming the text file

Once again we need to create at least one custom pipeline (receive pipeline) to process this flat-file document and translate it into is equivalent in XML.

To create the Receive Pipeline that will be responsible for processing and transforming the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Pipeline Files”, and then select the option “Receive Pipeline”, then provide the name you want to give the pipeline in this example: “FF_Employee_ReceivePipeline.btp”
• By selecting the option “Add”, the pipeline editor (BizTalk Pipeline Designer) will appear and that will let you view and add components to all the steps associated with receive pipeline: Decode, Disassemble, Validate and ResolveParty.

In this case, the pipeline that we will create, will be responsible for receiving a text file through a receive location and convert it to XML. For that, we will use the “Flat File Disassembler” component which is available in the Visual Studio Toolbox window (Ctrl+W, X).

• Drag it into the Pipeline in step “Disassemble”.

• Finally, select the component “Flat file disassembler”, go to its properties and on the “Document Schema” option: select the schema created earlier, in case the “PositionalTextFiles.FF_Positional_Employees”.

Note: If you want to create a send pipeline in order to transform an XML document into Flat File, we would follow the same steps, the difference is that we would have to drag the component “Flat File Assembler” on stage “Assemble”.

Once you have deployed your solution, you will want to ensure that you select the custom pipeline we just created in the Receive Locations that you expect to receive these flat files.

Now here it comes the same questions: Do you think the schema is entirely correct? Does it handles with all files accordingly to the expected results?

Or do we need to tune some of the properties of your schema… stay tuned to the next post.

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction
• BizTalk Server: Teach me something new about Flat Files (or not) – Files Delimited by Symbols
• BizTalk Server: Teach me something new about Flat Files (or not) – Delimited Flat File Schema Optimizations

You can download all the code used here from:

BizTalk Server: Dealing with Positional Flat Files (14 KB)
Microsoft | Code Gallery

Introduction

Transformations are one of the most common components in the integration processes. They act as essential translators in the decoupling between the different systems to connect. This article aims to help you understand the process of transforming a text file (also called Flat Files) into an XML document using BizTalk Server Flat File Schemas.

Normally we associate the transformations of documents to BizTalk maps, but the reality is that there are two types of transformations: structure transformation (semantics) and representation transformation (syntax). These latest occurs typically at receiving or sent ports of BizTalk Server,

This article intends to be an introductory note for whom is taking the first steps in this technology.

One of the most ancient and common standards for message representation is to use text files (Flat Files) like: CSV (Comma Separated Values) or TXT files, many of them custom-made for their systems. However, over the time, XML became the standard message format because of its widespread use by major corporations and open source development efforts. However do not be fooled and think that these kinds of message are outdated and rarely used, a good example is EDI messages, which is used extensively by large companies, so it is often necessary to transform text files into XML and vice versa.

While tools like Excel can help us interpret such files, this type of process is always iterative and requires few user tips so that software can determine where is a need to separate the fields/columns as well the data type of each field. But for a system integration (Enterprise Application Integration) like BizTalk Server, you must reduce any ambiguity, so that these kinds of operations can be performed thousands of times with confidence and without having recourse to a manual operator.

Map or Schema Annotation?

As mentioned in the introduction, we can characterize two types of transformations existing in BizTalk:

• Semantic Transformations: This type of transformation usually occurs only in BizTalk maps. Here, the document maintains the same syntax that is represented (XML) but changes its semantics (data content). This type of transformation is typically one-way, since that when we added and aggregate small parts of the information, that compose the document into another different document, we may miss important details for its reconstruction.

• Syntax Transformations: This type of transformations occurs in the receive or send pipelines and aim to transform a document into another representation, e.g. CSV to XML. Here, the document maintains the same data (semantics) but changes the syntax that is represented. I.e. we translate the document, but typically we don’t modify the structure. Normally, this type of transformation is bidirectional, since we still have the same semantic content, we can apply the same transformation logic and obtain the document in its original format. Common examples of these transformations are also conversions between HL7 and XML, or EDI and XML.

Note: In this article, we will talk only of Syntax transformations. If you are looking to learn more about semantic transformations, you can consult the article “BizTalk Server: Basics principles of Maps“.

How does the text files (Flat Files) are processed by BizTalk?

Internally, BizTalk “prefers” to use the message type XML. If messages are in XML format BizTalk “offers” numerous automatisms that are very useful in these environments, such as: message routing based on a particular field (promoted property); tracking and analysis of multidimensional values and dimensions with BAM (Business Activity Monitoring), or making logical decisions within orchestrations (business processes) using elements of the message.

If messaging is the foundation of BizTalk Server, the message schemas are the bedrock on which messaging is built. Fortunately, BizTalk supports the conversion of text files to XML in a simple and intuitive manner, using for that “Flat File Schemas” that are simple XML schemas (XSD) with specific annotations. At first glance, this may seem strange, because the XML Schemas (XSD) are used to describe XML files, however, BizTalk uses them as metadata to describe not only XML documents but also text files (flat file).

The trick is that all the necessary information, such as the delimiter symbols, or the element size in a positional file, i.e. the definition of the rules of parsing (transformation rules) are embedded in the form of annotations in XML Schema (XSD), thereby simplifying the reuse of all these schemes in different parts of the process. At any point, the document can be translated back into flat-file because the definition is declarative and symmetric.

Where the Syntax Transformations can occur?

This type of transformations can occur in receive or send pipelines, usually, text files (Flat Files) are processed at runtime as follows:

• The Flat Files are received by an adapter associated with a receive location (Folder in File System for example)
• A pipeline configured in the receive location will be responsible for transforming the Flat File into its equivalent XML;
• One or more interested in the message, such as an orchestration will subscribe to the XML document and this message will go through the business process. Note, in a pure messaging scenario there is no need to have orchestrations;
• If and when necessary, BizTalk can send XML messages, again, as text files (Flat Files) by using another pipeline in the send ports, which will be responsible for transforming the XML into its equivalent the Flat File;

As the image below shows:

The receive pipeline consists of four stages, being that syntax transformations may occur in two of them:

• Decode stage: This stage is used for components that decode or decrypt the message. The MIME/SMIME Decoder pipeline component or a custom decoding component should be placed in this stage if the incoming messages need to be decoded from one format to another. The syntax transformations can occur at this stage through a custom component.
• Disassemble stage: This stage is used for components that parse or disassemble the message. The syntax transformations should occur at this stage. In the example that will be demonstrated in this article, we will use the “Flat file disassembler” to transform a text file into XML.

• Validate stage: This stage is used for components that validate the message format. A pipeline component processes only messages that conform to the schemas specified in that component. If a pipeline receives a message whose schema is not associated with any component in the pipeline, that message is not processed. Depending on the adapter that submits the message, the message is either suspended or an error is issued to the sender.
• ResolveParty stage: This stage is a placeholder for the Party Resolution Pipeline Component.

Regarding send pipelines, they consist of three stages, being that syntax transformations may occur also in two of them:

• Pre-assemble stage: This stage is a placeholder for custom components that should perform some action on the message before the message is serialized.

• Assemble stage: Components in this stage are responsible for assembling or serializing the message and converting it to or from XML. The syntax transformations should occur at this stage.
• Encode stage: This stage is used for components that encode or encrypt the message. Place the MIME/SMIME Encoder component or a custom encoding component in this stage if message signing is required. The syntax transformations can occur at this stage through a custom component.

Related links

• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)
• BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

The post BizTalk Server: Transform text files (Flat Files) into XML–Introduction (Part 1) appeared first on SANDRO PEREIRA BIZTALK BLOG.

Necessary tools and artifacts

As mentioned earlier, to solve this problem – BizTalk Server: Transform text files (Flat Files) into XML – we must create at least two artifacts:

• Flat File Schema: with all the necessary information embedded in the form of annotations in XML Schema (XSD), such as the delimiter symbols, or the element size in a positional file, i.e. the definition of the rules of parsing (transformation rules). This type of artifact can be created manually or with the tool “BizTalk Flat File Schema Wizard”.
• Receive (and Send – optional) Pipeline: that will be responsible for processing and transforming the text file (Flat File) into its equivalent XML. This artifact can be created using the BizTalk Pipeline Designer.

BizTalk Flat File Schema Wizard

BizTalk Flat File Schema Wizard tool, integrated into Visual Studio, which allows us to easily and visually make the transformation of a text file (Flat File) into its equivalent XML representation. This tool supports two types of text files:

• Positional text files:

HEADERXXXXXXXXXXXXXXXXXXXXXXX
BODYXXXXXXXXXXXXXXXXXXXXXXXXX
BODYXXXXXXXXXXXXXXXXXXXXXXXXX
FOOTERXXXXXXXXXXXXXXXXXXXXXXX

Note: Header is in Portuguese “Cabeçalho”, therefore the text from the image is in Portuguese

• Or delimited by symbols:

1999990;1;P0110;1;1;20110307;
1999990;2;P0529;2;2;20110307;
1999990;3;P0530;3;3;20110307;

BizTalk Pipeline Designer

The editor of pipelines, BizTalk Pipeline Designer, allows us to create, visualize and edit pipelines; move pipeline components between the different stages and configure pipelines components.

This editor is integrated into Visual Studio and is mainly composed of 3 modules:

• Properties window: in this window, we can see and modify the properties of components in the different stages of the pipeline.
• Toolbox window: is used as a source for the design surface, it provides access to all the components that we use in pipelines.
• Design surface: where components from the Toolbox are dragged and dropped, which allows us to draw a graphical representation of a pipeline by inserting the components, available in the toolbox window, in the different stages of the pipeline.

Related links

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)
• BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

The post BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2) appeared first on SANDRO PEREIRA BIZTALK BLOG.

Constructing a Flat Files Schema – Practical example

For this project, we will use the BizTalk Server 2010 and Visual Studio 2010 and we will explain step by step what needs to be developed. Briefly, these are the steps we have to perform:

• Creating an instance of the text file that will serve as a test file for the project;
• Creating the Schema which will recognize the text file;
• Creating the Pipeline that will be responsible for processing and transforming the text file;
• Deploy the BizTalk Server solution;
• Configuring the BizTalk application;
• Run the solution;

The solution of this example, as well all the code, is available on MSDN Code Gallery: http://code.msdn.microsoft.com/BizTalk-Server-Transformar-0abe5767

We begin then by launching Visual Studio 2010 and create a new BizTalk project:

• “File –> New –> Project”, on BizTalk Projects, select the option “Empty BizTalk Server Project”;
• Insert the project name, physical location on the disk and the name of the solution.

Creating an instance of the text file that will serve as a test file for the project

Before we begin our development we need to create an instance or sample of the text file that will serve as a model for the creation of the Flat File Schema. Therefore we will create the following text file on our file system that will be used in our solution:

• Create a folder “<solution>\TESTFILES” where we will create/put the messages we want to transform. In this article we will use a text file delimited by symbols which will be composed of several lines with the following content:

Sandro;Pereira;1978-04-04;Crestuma;4415 Crestuma
Lígia;Tavares;1982-01-21;Seixo-Alvo;451 Seixo-Alvo
José;Silva;1970-09-19;Crestuma;4415 Crestuma
Rui;Barbosa;1975-09-19;Lever;4415 Lever

Each line consists of the following structure: Name (Nome), Surname (Apelido), Birthdate (Data Nascimento), Address (Morada) and Zip Code (Código Postal);
Note: in () is the equivalent in Portuguese found in pictures. The file “PESSOAS.txt” that we use for testing is available in the directory “<solution >\TESTFILES”.

• We will also create two folders that we will configure in the BizTalk Administration Console in order to test the solution.
  • Create a folder “<solution >\PORTS\IN” which will serve as the place of entry of Flat File files for conversion;
  • Create a folder “<solution >\PORTS\OUT” which will serve as the output location of the files after they have been converted.

Creating the Schema which will recognize the text file

To create the schema which will recognize the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Schema Files”, and then select the option “Flat File Schema Wizard”, then provide the name you want to give the scheme in this example: “TXT_to_XML.xsd”

• By selecting this option, we will be guided automatically by the tool “BizTalk Flat File Schema Wizard” that will help us to create a Flat File Schema and define its data structure (records, elements, attributes …) based on the text file specified. Select “Next” to continue.

• In the window “Flat File Schema Information” we will have to:
  • Select an instance of the text file that will serve as the model of the structure that we want to transform;
  • Although it is not necessary, it is good practice to rename the Record name “Root”. In this case, we will rename it to “People” (Pessoas)
  • And finally, assign a “Target namespace” to the scheme and define the encoding of the input file

• The wizard will load the text file so that we can begin to split it and map it into the desired structure. In this step, we need to define how the records or rows are differentiated. The structure of the example is:

Name;Surname;Birthdate;Address;Zip Code{CR}{LF}
(Nome;Apelido; Data Nascimento;Morada;Codigo Postal{CR}{LF})

Since each record “Person” (Pessoa) that we want to create is defined and contained in a line, in the “Select Document Data” we will select all the data portion of the document that will set the record, i.e. the whole first line.

• In the window “Select Record Format” we will define whether we are dealing with a Flat File Delimited by symbols or is positional, in our case we will select the “By delimiter symbol” which is delimited by a return Carriage Return/Line Feed.

• In the window “Delimited Record” we will provide the record delimiter, in this case as we want to define the structure of person (Pessoa), i.e. each row is a person, our limiter is {CR}{LF} (Carriage Return/Line Feed)

• In the window “Child Elements” we will define what kind of element we want to assign to the registry. As we are defining the Person structure and the file contains multiple people, we have to select the “Element Type” as “Repeating record.” If we do not perform this step, we will not have the ability to split the record into multiple elements/attributes individual.

• At this stage we successfully created the record Person (Pessoa), i.e. we have just map that each line of the text file corresponds to a record Person. In the “Schema View” select “Next” to continue processing the message

• At this stage, the wizard restarts the whole process described above, but if you noticed, the wizard no longer selects all information contained in the text file, but only what was selected to define the record Person. What we will do now is split the information of record “Person” in different elements, for that, we will select only the information required leaving out the Carriage Return/Line Feed.

• Once again our structure is delimited by symbols (;), then we will select the option “By delimiter symbol”

• As we can analyze all the elements are separated by the semicolon (;) that is our delimiter, then in the window “Delimited Record” we must change the value of the “Child delimiter” option to “;”.

• In this window “Child Elements”, we will define the different elements/attributes of the structure of the record person. This operation is very similar to any XSD, where we can define the different names and data types. Adjust the values according to the image:

• Finally, the wizard will show the equivalent XML structure that your text file document will have. Once you select the option “Finish”, the scheme will be available for you to use in your BizTalk solution.

After we finalize the creation of the Flat File Schema which will contain the transformation rules of the text file, we can easily test our transformation, without having to get out of our development tool (Visual Studio) and without having to publish our solution.

If we select the Flat File Schema that we just created and access to its properties, we can see that by default all properties are preconfigured so that we can perform tests to our message transformation: the input instance file is configured with the file which was used to create the schema; and the correct input (Validate Instance Input Type: Native) and output (Generate Instance Output Type: XML) formats.

To test, you simply select the schema and with the right mouse button select the option “Validate Instance”:

This option will use the configured file and validate all transformation rules defined and subsequently present the final result or occurring errors:

Creating the Pipeline that will be responsible for processing and transforming the text file

To create the Receive Pipeline that will be responsible for processing and transforming the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Pipeline Files”, and then select the option “Receive Pipeline”, then provide the name you want to give the pipeline in this example: “ReceivePipelineCustomPessoas.btp”

• By selecting the option “Add”, the pipeline editor (BizTalk Pipeline Designer) will appear and that will let you view and add components to all the steps associated with receive pipeline: Decode, Disassemble, Validate and ResolveParty. In this case, the pipeline that we will create will be responsible for receiving a text file through a receive location and convert it to XML. For that, we will use the “Flat File Disassembler” component which is available in the Visual Studio Toolbox window (Ctrl+W, X). Drag it into the Pipeline in step “Disassemble”.

• Finally, select the component “Flat file disassembler”, go to its properties and on the “Document Schema” option: select the schema created earlier, in case the “TXT_to_XML.”.

Note: If you want to create a send pipeline in order to transform an XML document into Flat File, we would follow the same steps, the difference is that we would have to drag the component “Flat File Assembler” on stage “Assemble”.

Deploy the BizTalk Server solution

All artifacts created yet must be installed on BizTalk Server 2010. However, before we can deploy the solution there are some settings we need to do or guarantee:

• Before deploy the Visual Studio solution into a BizTalk application is necessary that all project assemblies are signed with a strong name key, since they must be installed in the global assembly cache (GAC). For this we need:
  • In Visual Studio Solution Explorer, right-click on the project name and select “Properties” option
  • Select the “Signing” tab and choose “New” in the drop-down box “Choose a strong name key file”

• Assign a name, for example “TXTtoXML.snk”

• Likewise, before deploying the Visual Studio solution into a BizTalk application, we need to first define the BizTalk deployment properties, especially the properties of the “BizTalk Group”. If the Visual Studio solution contains multiple projects, we need to set the properties for each project separately.
  • In Visual Studio Solution Explorer, right-click on the project name and select “Properties” option
  • Select the “Deployment” tab and configure the name we want to assign to the BizTalk application in the property “Application Name”: in our example “TransfFlatFilesEmXML Demo”. The remaining properties may stay with the default values.

• To learn more about these properties go to http://msdn.microsoft.com/en-us/library/aa577824.aspx

Finally, we can Build and Deploy the project so that it is published as a BizTalk application inside BizTalk Server:

However, in order to publish a BizTalk solution, you must be a member of the group “BizTalk Server Administrators”. If the option “Install to Global Assembly Cache” is enabled, on the “Deployment” properties, then you also need permissions to read/write on GAC.

Related links

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

The post BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3) appeared first on SANDRO PEREIRA BIZTALK BLOG.

Configuring the BizTalk application

This is the final step of this process. In order to properly test the solution we have been developing in BizTalk Server, we need to configure the application that was created in the publication.

For this, we will access the BizTalk Server Administration console to create and configure the following artifacts:

• One receive port and location to process the input text files;
• One send port to save the processed files in XML format…

Open the BizTalk Server Administration console under “Start –> Programs –> Microsoft BizTalk Server 2010”. On the left tree menu, maximize “BizTalk Server Administration –> BizTalk Group (…) –> Applications”, and look for the application called “TransfFlatFilesEmXML Demo” and maximize also this application.

• Press the right button on top of “Receive Port” menu and select “New –> One-Way Receive Port …” option;
• A new window appears allowing us to define the port properties:
  • In the “General” tab, set the name of the receive port: “ReceiveTXT”.

• In the “Receive Locations” tab, select “New” and set the name to your receive location: “ReceiveTXT_to_XML”; in the property “Type” select the option FILE; and in the property “Receive pipeline” select from the drop down box the pipeline we created earlier: “ReceivePipelineCustomPessoas”;

• On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set “Receive Folder” property, with the folder you created earlier “<solution>\PORTS\IN”. On property “File Mask” put the following value: “*. txt”, setting that will be processed only files with the extension. “txt”. Finally, select the “Ok” button.

• To finish the process of creating the receive port press “Ok”.

The above steps mentioned were needed to create one Receive Port and its Receive Location. Now we will create a Send Port to save messages on the file system after they are processed by BizTalk:

• Press the right button on top of “Send Ports” menu and select “New –> Static One-way Send Port…”
• A new window appears allowing us to define the port properties:
  • In the “General” tab, set the name of the send port: ” SaveXMLFile “
  • In the property “Type” select the option FILE;
  • In the property “Send Pipeline” set the pipeline to “XMLTransmit”. This is a native pipeline that is used for processing XML messages.

• On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set the “Destination Folder” property, with the folder you created earlier “<solution>\PORTS\OUT”. On property “File Mask” put the following value: “%MessageID%.xml”, this is a special tag (BizTalk Macro) that will set the name of each file written on the file system with the unique identifier of the message. Finally, select the “Ok” button to return to the previous window.

• In order for the Send Port subscribes to all files, we need to set a filter on the tab “Filters” with the following expression: “BTS.ReceivePortName == ReceiveTXT”.
  • Note: This setting will force each message in the MessageBox which has originated from the Receive Port: ReceiveTXT, be routed and processed by the Send Port we are finished creating.

• To finish the process of creating the send port press “Ok”.

Finally, we need only to start our application. To do that press right button on top of the application name “TransfFlatFilesEmXML Demo” and select “Start …”.

Testing the solution

To test the BizTalk application we only need to copy a text file to the directory that is configured on the receive location: “<solution>\PORTS\IN”. The result should appear in the form of an XML document in the folder configured in the send port: “<solution>\PORTS\OUT”.

Note: The file disappears automatically from the input folder, so you should avoid doing drag & drop (move) with the test files at the risk of not being able to use it again.

Conclusion

As presented in this article, BizTalk Server helps us to solve many of the existing problems concerning message transformation: semantic and syntax Transformations, only with “out of the box” features within the product, in few steps and most of the time without developing any code.

The fact of all the definitions of the rules of parsing (transformation rules) are embedded in the form of annotations in XML Schema (XSD), such as the delimiter symbols, or the element size in a positional file, simplifies the reuse of all these schemes in different parts of the process. At any point, the document can be translated back into flat-file because the definition is declarative and symmetric, in contrast to what might happen with a “normal” programming language like C#, where we would have to create two components to translate text to XML and vice versa.

The model used by BizTalk schemas, centralizes all the message definitions in a single artifact, simplifying the maintenance and evolution of the specification and therefore simplifying future reuse and dependencies management. When we set up this type of project, is exactly this kind of guarantees that keeps the systems operating properly for many years, and at the same time, they make life easier to those who one day have to correct and/or develop any of these processes

Related links

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)

Source code

The solution of this example, as well all the code, is available on MSDN Code Gallery:
BizTalk Server – Transform text files (Flat Files) into XML (64.5 KB)
MSDN Code Gallery

The post BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4) appeared first on SANDRO PEREIRA BIZTALK BLOG.

Following one of my last speaking engagement in the Porto.Data Community Meeting about Flat Files: How to process Flat Files documents (TXT, CSV …) in BizTalk Server where I address some of the following topics:

• How to process Flat Files documents (TXT, CSV …) in BizTalk Server.
• What types of flat files are supported?
• How is the process of transforming text files (also called Flat Files) into XML documents (Syntax Transformations)
• Where does it happen and which components are needed
• Or how can I perform a flat file validation?
• And so on

I decided to return writing about this topic on my blog, a topic that I already wrote in the past describing and explaining almost all of the topics mention above:

• BizTalk Server: Transform text files (Flat Files) into XML – Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)
• BizTalk Server: Transform text files (Flat Files) into XML – A practical example (Part 3)
• BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

But this time transforming my previous session in a sequence of blog posts where I will address and explain how to solve some of the typical or common flat files structures and scenarios.

The simplest ones:

• Text Files Delimited by symbols
  • For example:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4400
José;Silva;1972;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

• Where we have: Name,Surname,Birthyear,Address,ZipCode
• Positional Text Files
  • For example:

0120200300 01 PT        Sandro      Pereira         Rua Central, 133      Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        José        Silva           Crestuna              Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        Rui         Barbosa         Porto                 Porto       Porto              4400
0120200300 01 PT        Miguel      Silva                                 Porto       Porto              4415 Crestuma

• Where we have:

GroupId|PrimaryKey|SecondaryKey|ControlFlag|PartyNo|Remark|Country|Name|Surname|Street|City|District|ZipCode

0120|20|03|0|0 |01 |PT |Sandro |Pereira |Rua Central, 133 |Crestuma |Vila Nova de Gaia |4415 Crestuma

And evolve to more challenges scenarios like:

• Handling Record Tag Identifiers, where we will treat each Tag Identifier differently
  • For example:

PO1999-10-20
USERSandro Pereira,1978-04-04, Crestuma
USERCelso Pereira,1978-04-04, Crestuma
ITEM872-AA|Super Bock|Green
ITEM871-AA|Super Bock|Black

• Where we have:
  • Header
    • Date
  • User
    • Name|Birthyear|Address
  • Item
    • Id|Product|Specification
• Dealing with Headers and Trailers
  • For example: (see the original post here: Walkthrough: Flat File Disassembly Using a Header and Trailer)

East Coast Facility
ERROR102|0|High|Sprocket query fails.|1999-05-31T13:20:00.000-05:00
ERROR16502|2|Low|Time threshold exceeded.|1999-05-31T13:20:00.000-05:00
8675309

• Where we have:
  • Location
  • ERROR
    • ID|Type|Priority|Description|DateTime
  • BatchID
• Or simple How can I remove a simple header from a Text File
  • For example

P_ABB,NUMBER,SHIP_ID,CREATE_DATE,SHIP_NO,SHIP_DATE,SHIP_DEST,PO_NO,LOT_TYPE,LOT_NO,SHIP_QTY,SHIP_PRD_NO,OBS_LOT
PA1,IN1900011,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PD806-D0,AGT;AKT;LOOP
PA1,IN1900012,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PM806-D0.0J,AGT;AKT;FLAT

• Where the structure of the message is described in the header (first line) – This scenario, for example, is very common in CSV files.

Until we reach and address an advanced topic like:

• Debatching Flat Files messages
• Perform Deep Validation in flat files processes
• Create generic Receive and Send Pipelines and how to configures this pipelines through BizTalk Administration Console
• And so on

And hopefully teaching you something new (trick or tip) about Flat Files or at least trying to demystify some issues related with Flat Files.

Most of this things will be solved by explaining to you how to create, for each scenario, flat file schema from flat file document instances using the BizTalk Flat File Schema Wizard.

Others, how can we tuning some properties afterward so that we can have the desired output.

Let’s the fun begin.

The post BizTalk Server: Teach me something new about Flat Files (or not) – Introduction appeared first on SANDRO PEREIRA BIZTALK BLOG.

This post is intended to consolidate some basic principles about Flat File Schema Wizard

First of all, BizTalk Flat File Schema Wizard is a template inside Visual Studio that allows you to convert Flat File into XML message, which BizTalk can understand, in a graphical a simple manner. That basically supports two types of text files: Positional text files and/or delimited by symbols. See more:

• BizTalk Server: Transform text files (Flat Files) into XML–Introduction (Part 1)
• BizTalk Server: Transform text files (Flat Files) into XML – Necessary tools and artifacts (Part 2)

The following walkthrough shows you how to create a flat file schema from semicolons separate flat file document that stores people information using the BizTalk Flat File Schema Wizard.

File sample:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4400
José;Silva;1972;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

You can see that we have, in this example, four records with five fields specifying Name, Surname, Birth year, Address and Zip code of each person.

Now I will walk you through the required steps at design time:

• using the Flat File Schema Wizard to generate the Flat File Schema;
• and the pipeline editor to create a custom pipeline;

To enable the runtime to support the processing of the Flat File described earlier (People)

We begin then by launching Visual Studio and create or open a BizTalk project:

• “File –> New –> Project”, on BizTalk Projects, select the option “Empty BizTalk Server Project”;
• Insert the project name, physical location on the disk and the name of the solution.

Using Flat File Schema Wizard to generate a schema based on a file delimited by Symbols

To create the schema which will recognize the ‘people’ text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”

• On “Installed Templates” menu in the window “Add New Item”, select the option “Schema Files”, and then select the option “Flat File Schema Wizard”, then provide the name you want to give the scheme in this example: “FF_DSYMBOL_PEOPLE.xsd”

By selecting this option, we will be guided automatically by the tool “BizTalk Flat File Schema Wizard” that will help us to create a Flat File Schema and define its data structure (records, elements, attributes …) based on the text file specified.

• Click “Next” to continue.

• In the window “Flat File Schema Information” we will have to:
  • Select an instance of the text file that will serve as the model of the structure that we want to transform;
  • Although it is not necessary, it is good practice to rename the Record name “Root”.
    • In this case, we will rename it to “People”
  • And finally, assign a “Target namespace” to the scheme and define the encoding of the input file
  • Click “Next” to continue.

Note: The wizard will load the text file so that we can begin to split it and map it into the desired structure. In this step, we need to define how the records or rows are differentiated. The structure of the example is: Name;Surname;Birthdate;Address;Zip Code{CR}{LF}

• Since each record “Person” that we want to create is defined in a single line, in the “Select Document Data” we can select only the data portion of the document that will set the record, i.e. the whole first line. Click “Next” to continue.

Note: Alternatively, let’s call it option 2, we could select all the data of the file, however with this approach we need to treat differently the data a few steps ahead in the “Child Elements” window.

• In the window “Select Record Format” we will define whether we are dealing with a Flat File Delimited by symbols or is positional, in our case we will select the “By delimiter symbol” because each record is delimited by a return Carriage Return/Line Feed ({CR}{LF}). Click “Next” to continue.

• In the window “Delimited Record” we will provide the record delimiter, in this case as we want to define the structure of person, i.e. each row is a person, our limiter is {CR}{LF} (Carriage Return/Line Feed). The “Child delimiter” combo box include by default the following options: {CR}, {LF}, {TAB}, {SPACE}, {0x1A}, {}, {.}, {;} and {CR}{LF}, however you can provide your own symbol.
  • Click “Next” to continue.

• In the window “Child Elements” we will define what kind of element we want to assign to the registry. As we are defining the Person structure and the file contains multiple people, we have to select the “Element Type” as “Repeating record” If we do not perform this step, we will not have the ability to split the record into multiple elements / attributes individual.
  • Note: If you do not define the element as “Repeating record” (or “Record”), you will not have the ability to break down your record into individual elements or attributes.
  • Click “Next” to continue.

Note: earlier I told that we could select all the data inside the flat file and that we would need to treat the information differently in this step – option 2. So if you select this option, it will present all the lines in the document in this window, because we want to have an “Array” of Persons, in this step we need to select the “Element Type” of the first element as “Repeating record” and the rest of the set as “Ignore” (You set this to Ignore because you have already specified Repeating Record for the Person element)

In this point, I have defined how the element Record looks like but I have not broken down the various elements that make up the record itself. We have just map that each line of the text file corresponds to a record Person.

• In the “Schema View”, click “Next” to continue processing the message. Click “Next” to continue.

In this phase, the wizard restarts the whole process described above, but if you noticed, the wizard no longer selects all information contained in the text file, but only what was selected to define the record Person.

• What we will do now is split the information of record “Person” in different elements, for that, we will select only the information required leaving out the Carriage Return/Line Feed. Click “Next” to continue.

• Once again our structure is delimited by symbols (;), then we will select the option “By delimiter symbol” and then c“Next” to continue.
• As we can analyze all the elements are separated by the semicolon (;) that is our delimiter, then in the window “Delimited Record” we must change the value of the “Child delimiter” option to “;”.

• In this window “Child Elements”, we will define the different elements/attributes of the structure of the record person. This operation is very similar to any XSD, where we can define the different names and data types. Adjust the values according to the table below:

Note: You should rename the parent node, each record, element or attribute in the BizTalk Flat File Schema Wizard, mainly in this step by modify the “Element Name” field to your desired name, however if you forgot to do that or if you make a mistake defining one of them, you can always rectify it in the Schema Editor. The same applies to the Data Type.

• Click “Next” to continue.
• Finally, the wizard will show the equivalent XML structure that your text file document will have. Once you select the option “Finish”, the scheme will be available for you to use in your BizTalk solution.

After we finalize the creation of the Flat File Schema which will contain the transformation rules of the text file, we can easily test our transformation, without having to get out of our development tool (Visual Studio) and without having to deploy our solution.

If we select the Flat File Schema that we just created and access to its properties, we can see that by default all properties are preconfigured so that we can perform tests to our message transformation:

• The input instance file is configured with the file which was used to create the schema;
• And with the correct input and output formats
  • Validate Instance Input Type: Native
  • Generate Instance Output Type: XML

To test, you simply right-click in the schema that we just created and select the option “Validate Instance”:

This option will use the configured file to execute and validate all transformation rules defined in the Flat File Schema and present the final result or occurring errors in the “Output” window

Did you know that: Delimited fields in flat files have a limit of 50000000 characters

Creating the Pipeline that will be responsible for processing and transforming the text file

As I explained in detail in one of my previous posts: BizTalk Server: Transform text files (Flat Files) into XML–Introduction (Part 1), the Syntax Transformations occur in the Disassembling and Assembling stages within the receive and send pipelines, having this Flat File schema will not do us much good until we create the pipelines to provide the translation between Flat Files and XML

To create the Receive Pipeline that will be responsible for processing and transforming the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Pipeline Files”, and then select the option “Receive Pipeline”, then provide the name you want to give the pipeline in this example: “FF_People_ReceivePipeline.btp”

• By selecting the option “Add”, the pipeline editor (BizTalk Pipeline Designer) will appear and that will let you view and add components to all the steps associated with receive pipeline: Decode, Disassemble, Validate and ResolveParty.

In this case, the pipeline that we will create it will be responsible for receiving a text file through a receive location and convert it to XML. For that, we will use the “Flat File Disassembler” component which is available in the Visual Studio Toolbox window (Ctrl+W, X).

• Drag it into the Pipeline in step “Disassemble”.

• Finally, select the component “Flat file disassembler”, go to its properties and on the “Document Schema” option: select the schema created earlier, in case the “FF_DSYMBOL_PEOPLE”.

Note: If you want to create a send pipeline in order to transform an XML document into Flat File, we would follow the same steps, the difference is that we would have to drag the component “Flat File Assembler” on stage “Assemble”.

Once you have deployed your solution, you will want to ensure that you select the custom pipeline we just created in the Receive Locations that you expect to receive these flat files.

Takeaway

Now the big questions here is: Do you think the Schema is entirely correct? Does it handle all files accordingly to the expected results?

Or do we need to tune some of the properties of your schema… stay tuned to the next post.

Some references:

• Flat File Messages with Delimited Records
• How to Configure the Flat File Disassembler Pipeline Component
• BizTalk Flat File Schema Wizard

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction

You can download all the code used here from:

BizTalk Server: Dealing with Files Delimited by Symbols (71,9 KB)
Microsoft | Code Gallery

The post BizTalk Server: Teach me something new about Flat Files (or not) – Files Delimited by Symbols appeared first on SANDRO PEREIRA BIZTALK BLOG.

We ended the last post asking: Do you think the Schema is entirely correct? Does it handle all files accordingly to the expected results? Or do we need to tune some of the properties of your schema…

When we start working with Flat File schemas for BizTalk, we may think that just using the Flat File wizard is going to be enough. Once the schema is created, we just need to create a custom pipeline and add the schema to the Document Schemas of a Flat File disassembler and it will be good… well let me tell you that most probably is not enough and the reality may be very different.

A common mistake that we sometimes do is to rely on a single sample message to generate/validate our Flat File Schemas. And we ended up getting possibly several different errors in production due to unforeseen cases that end up not being supported in our current schema version.

So it is real important to get all the information, as possible, from the different teams, for example:

• Are there any optional elements?
• If so, will they be delete from the message? Or will they be present without any content (blank/null)
• Does the message will always end with a blank line? i.e., does the last line contain a Carriage Return/Line Feed (CR+LF)
• What are the Data type of the elements?

And many, many more….

The following walkthrough shows you some of the common scenarios that you can expect and how you can tune some of the properties of our Flat File Schemas to deal with them.

Cosmetic operations

While generating the Flat File Schema with the Flat File Schema Wizard you should always provide/set:

• The correct or proper namespace for the schema
• Rename the parent node, each record, element or attribute
• And set the correct data type of each element or attribute

However if you forgot to do that or if you make a mistake defining one of them, you can always easily rectify it in the Schema Editor.

I call it cosmetic operations because they don’t infer “any changes” in the translation rules generated by the wizard, they only changed the way the elements are presented in the message, if you change the name of the element or attributes. Or they will have effect in the validation of the message, if we change the data types.

You can accomplished that by:

• Open the schema in BizTalk Editor.
• Change the Schema Target Namespace
  • In the BizTalk Editor dialog, right-click the <Schema> node in the tree view and click Properties. The Properties dialog appears.
  • In the Properties dialog, the Target Namespace displays under the General category for the <Schema> node.
  • In the Target Namespace field, type the name of your new target namespace and click OK.
• Change the Field Attribute or Field Element Name
  • Select the Field Attribute or Field Element node and then right-click and select the “Rename” option
  • Type the desired Field Attribute or Field Element name and press “Enter”
• Change the Field Attribute or Field Element Data Type
  • Select the Field Attribute or Field Element node and then right-click and select the “Properties” option
  • In Properties window, set the “Data Type” property to the desired type

Validating Flat Files Schemas through the Visual Studio with different scenarios

Let us forget the runtime for now, and focus firstly on the development phase where we can test and validate our schemas directly from Visual Studio as explained, once again, in the previous post.

Dealing with Files ending with a newline or that doesn’t ends with a newline

Initially we had the following example:

Sandro;Pereira;1978;Crestuma;4415{CR}{LF}
Vânia;Braziela;1981;Vila Nova de Gaia;4400{CR}{LF}
José;Silva;1972;Crestuma;4415{CR}{LF}
Rui;Barbosa;1975;Lever;4415{CR}{LF}

Which was the base to generate our Flat File schema: “FF_DSYMBOL_PEOPLE.xsd”.

If you paid attention, while we were generating our Flat File Schema, that in the “Select Document Data” window of the BizTalk Flat File Schema Wizard, you can notice that the last line contains a Carriage Return/Line Feed (CR+LF), which means that if you open the file for example in Notepad, you will notice that the file ends with a blank line.

Note: You can check the file “FF_DSYMBOL_PEOPLE.txt” that is available with the source code.

So let us try with a simple change by removing the Carriage Return/Line Feed so that the files does not end with a newline:

Sandro;Pereira;1978;Crestuma;4415{CR}{LF}
Vânia;Braziela;1981;Vila Nova de Gaia;4400{CR}{LF}
José;Silva;1972;Crestuma;4415{CR}{LF}
Rui;Barbosa;1975;Lever;4415

Check the file: “FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt”

If we now try this file instance against our Flat File Schema by:

• In Solution Explorer, right-click the schema that you want to validate, and then click Properties.
• In Properties window for the schema, for Input Instance Filename enter the name and path for the “FF_DSYMBOL_PEOPLE
  • Check more about Schema File Properties here.
• WithoutEnterLastLine.txt” message instance, or browse to the file, select it, and then click OK.
• Right-click the schema that you want to validate, and then click Validate Schema

You will notice that we will get the following error:

Invoking component…
<path>\DemoFile\FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt: error BEC2004: Unexpected data found while looking for:
‘\r\n’
The current definition being parsed is Person. The stream offset where the error occured is 133. The line number where the error occured is 4. The column where the error occured is 23.
<path>\FF_DSYMBOL_PEOPLE.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE.xsd, file: <file:///path\FF_DSYMBOL_PEOPLE WithoutEnterLastLine.txt>.
Component invocation succeeded.

If, on the other hand, we had originally generated the Flat File Schema using the message instance without the Carriage Return/Line Feed at the end.

And if we know try it with an instance containing a blank line at the end of the message we would receive a similar but different error:

Invoking component…
<path>\FF_DSYMBOL_PEOPLE.txt: error BEC2004: Unexpected end of stream while looking for:
‘;’
The current definition being parsed is People. The stream offset where the error occured is 139. The line number where the error occured is 5. The column where the error occured is 0.
<path>\FF_DSYMBOL_PEOPLE_Advance.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE_Advance.xsd, file: <file:///path\FF_DSYMBOL_PEOPLE.txt>.
Component invocation succeeded.

I’ve seen situations/solutions where there were created pipeline components to address and solve these type of problems, which isn’t the best and the proper solution!

In both cases the solution for solving these problems is the same: ensure that the “Allow Message Breakup At Infix Root” property on the Schemas node is set to “true” by:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, right-click in the Schema node and select the Properties option
• In Properties window for the schema, set the “Allow Message Breakup At Infix Root” property to “true”

After making this change, if we try to validate the two previous messages instances, we will notice that now everything will work perfectly.

So if you notice in the properties of our Schema:

• The “Child Order” property of the People Root Node is set as: “Postfix”
• And the “Child Order” property of the “Person” record is set as: “Infix”

What does it means? Child Order must always have a value. The possible values are:

• Prefix: Specifies that the delimiter appears before the data that it is delimiting.
• Postfix: Specifies that the delimiter appears after the data that it is delimiting.
• Infix: Specifies that the delimiter appears between different items of data, resulting in one less delimiter than when Prefix or Postfix is chosen.
• Default Child Order: Sets the child_order attribute to “default”, specifying that the child order specified using the Default Child Order property should be used.
• And Conditional Default: Removes the corresponding annotation in the XSD representation of the schema, if any, resulting in the following behavior:
  • If the Record node has a tag identifier, then the child order will be prefix.
  • Otherwise, the child order will be infix.

There are two scenarios related to delimited flat files for which special considerations apply when setting the Child Order property.

• The first such scenario concerns situations in which the flat file document has a header, a body, and optionally, a trailer.
• The second scenario is that this property must be set according to what the runtime components expect for the nodes

In our instance message, the data/delimiter sequence in the conceptual loop would merely be one or more person records. Of course, only in the case where there are more than one person records would there be a delimiter to separate them. In that case, the number of delimiters is one less than the sets of things being delimited, and the delimiters are located between the delimited items in a structure known as Infix.

• Number delimiter = Number Records – 1

Which is not the case in all our messages instances. Our first instance have the same number of “valid records” and delimiters.

By setting “Allow Message Breakup At Infix Root” property to “true”, this tells the parser whether to allow message breakup, and will ensure that the Infix stream is processed properly by the FF Disassembler.

Dealing with Optional Elements

Using Suppress Empty Nodes property

So let’s imagine a different scenario, this time we will deal with optional elements. In our first scenario while discussing with our time we realize that the element “Birthyear” was appearing empty in several messages, and it should be considered as an optional element:

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

By default, we need to have a complete instance of the message, for us developers be able to create all the transformation rules, which means that by default, when we finish creating the Flat File Schema through Flat File Schema Wizard tool, none of the records, elements or attributes are optional.

All of these changes/optimizations/configurations, they have to be made after the Flat File Schema been created.

You can always modify an existing Flat File schema that was created by the wizard (name of the elements, data type and so on). So to accomplish that we need to:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, expand People root node, expand Person record and right-click in the Birthyear element and select the Properties option
• In Properties window, set the “Min Occurs” property to “0”

If we now try to validate the previous message instance (“FF_DSYMBOL_PEOPLE WithOptional.txt”), we ended up receiving another error:

Invoking component…
Validation generated XML output <file:///<path>\FF_DSYMBOL_PEOPLE_Advance_output.xml>.
<path>\FF_DSYMBOL_PEOPLE_Advance_output.xml: error BEC2004: The ‘Birthyear’ element is invalid – The value ” is invalid according to its datatype ‘http://www.w3.org/2001/XMLSchema:int’ – The string ” is not a valid Int32 value.
<path>\FF_DSYMBOL_PEOPLE_Advance.xsd: error BEC2004: Validate Instance failed for schema FF_DSYMBOL_PEOPLE_Advance.xsd, file: file:///path\FF_DSYMBOL_PEOPLE WithOptional.txt.
Component invocation succeeded.

And of course the reason is obvious, an empty string is not a valid integer.

Making this element nillable doesn’t make any difference, because in both situations the parser will continue to add an empty element:

<Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear></Birthyear>
    <Address>Crestuma</Address>
    <ZipCode>4415</ZipCode>
</Person>

To solve that situation there are a few additional flat file node properties that we can use. You can see the full list here:

• Schema Node Properties

And in our case we are looking for the “Suppress Empty Nodes” property (or suppress_empty_nodes). By setting this property to “True” it will remove all the empty nodes from the XML stream.

• This can be used to eliminate fields that are empty after being parsed, but the XSD type doesn’t allow empty values.

To accomplish that we need to:

• In BizTalk Editor, right-click in the Schema node and select the Properties option
• In Properties window for the schema, set the “Suppress Empty Nodes” property to “true”

After making this change, if we try to validate the previous message instance, we will notice that now everything will work perfectly.

Note: When using Delimited files, the optional element in the beginning or in the middle of the each line (record node) need always to be present in the text with empty values (;;) otherwise the parsing rules will fail or will perform an incorrect translation of the message.

If the “Birthyear” is set as string and we receive this type of messages

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;Crestuma;4415
Rui;Barbosa;1975;Lever;4415

The message will be translated incorrectly and there is nothing you can do about it, the end result would look like this:

  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>Crestuma</Birthyear>
    <Address>
      4415
      Rui
    </Address>
    <ZipCode>Barbosa;1975;Lever;4415</ZipCode>
  </Person>

Dealing with Optional Elements (last element with the data type string)

Continue in the same scenario/message type, after a further series of tests, we noticed another unforeseen situation. Once again several messages were being executed incorrected because some of the element were not being parsed the element “ZipCode” – note that this element is a string

Sandro;Pereira;1978;Crestuma;4415
Vânia;Braziela;1981;Vila Nova de Gaia;4300
José;Silva;1972;Crestuma
Rui;Barbosa;1975;Lever;4415

Note: Contrary to what happens with elements at the beginning and end of the message, if the last element is an optional element, then it is not required to appear in the text (see line 3 – it has one element less)

We don’t receive any error message in BizTalk Server runtime but we notice that we messages are being integrated with incorrect data. After further analysis, let’s take the sample above, we notice that the message are being translated incorrectly and the last node is being added to the “ZipCode” element and not in a different Person node as supposed

  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>1972</Birthyear>
    <Address>
      Crestuma
      Rui
    </Address>
    <ZipCode>Barbosa;1975;Lever;4415</ZipCode>
  </Person>

To solve this problem we once again need set the element, in this case “ZipCode” as optional:

• Double-click the schema to open the item in the Schema Editor;
• In BizTalk Editor, expand People root node, expand Person record and right-click in the ZipCode element and select the Properties option
• In Properties window, set the “Min Occurs” property to “0”

<People xmlns="http://DelimitedBySymbols.FF_DSYMBOL_PEOPLE_Advance">
  <Person xmlns="">
    <Name>Sandro</Name>
    <Surname>Pereira</Surname>
    <Birthyear>1978</Birthyear>
    <Address>Crestuma</Address>
    <ZipCode>4415</ZipCode>
  </Person>
  <Person xmlns="">
    <Name>Vânia</Name>
    <Surname>Braziela</Surname>
    <Birthyear>1981</Birthyear>
    <Address>Vila Nova de Gaia</Address>
    <ZipCode>4300</ZipCode>
  </Person>
  <Person xmlns="">
    <Name>José</Name>
    <Surname>Silva</Surname>
    <Birthyear>1972</Birthyear>
    <Address>Crestuma</Address>
  </Person>
  <Person xmlns="">
    <Name>Rui</Name>
    <Surname>Barbosa</Surname>
    <Birthyear>1975</Birthyear>
    <Address>Lever</Address>
    <ZipCode>4415</ZipCode>
  </Person>
</People>

There is a Flat File Schema property call “allow_early_termination” (Early Terminate Optional Fields) that is a Boolean property. The use of this property is this situation is incorrect and does produce any changes in outcome. We will address and explain this property at another time.

Some references:

• Document Validation in the Flat File Disassembler Pipeline Component
• Schema Node Properties
• Additional Flat File Properties

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction
• BizTalk Server: Teach me something new about Flat Files (or not) – Files Delimited by Symbols

You can download all the code used here from:

BizTalk Server: Dealing with Files Delimited by Symbols (96 KB)
Microsoft | Code Gallery

The post BizTalk Server: Teach me something new about Flat Files (or not) – Delimited Flat File Schema Optimizations appeared first on SANDRO PEREIRA BIZTALK BLOG.

In the second post of this series, we follow a simple walkthrough explaining the basic principles to create a flat file schema from a file delimited by symbols. Now it’s time to do the same thing, but this time translating an inbound positional flat file.

A positional flat file it’s normally a file that has fields that have a fixed maximum length and records that have a common end-of-record terminator, for example:

Sandro      Pereira         {CR}{LF}
José        Silva           {CR}{LF}

Where the first fixed 12 characters describe the Name field and the following 16 characters describe the Surname field. White space padding the end of each field and the Carriage Return/Line Feed characters is the common end-of-record terminator.

The following walkthrough shows you how to create a flat file schema from a positional flat file document that stores employee information using the BizTalk Flat File Schema Wizard.

File sample:

0120200300 01 PT        Sandro      Pereira         Rua Central, 133      Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        José        Silva           Crestuna              Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        Rui         Barbosa         Porto                 Porto       Porto              4400 Crestuma
0120200300 01 PT        Miguel      Silva                                 Porto       Porto              4415 Crestuma

Once again I will walk you through the required steps at design time:

• using the Flat File Schema Wizard to generate the Flat File Schema;
• and the pipeline editor to create a custom pipeline;

To enable the runtime to support the processing of the Flat File described earlier (Employees)

We begin then by launching Visual Studio and create or open a BizTalk project:

• “File –> New –> Project”, on BizTalk Projects, select the option “Empty BizTalk Server Project”;
• Insert the project name, physical location on the disk and the name of the solution.

Using Flat File Schema Wizard to generate a schema based on a positional flat file

To create the schema which will recognize the ‘Employees’ text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”
• On “Installed Templates” menu in the window “Add New Item”, select the option “Schema Files”, and then select the option “Flat File Schema Wizard”, then provide the name you want to give the scheme in this example: “FF_Positional_Employees.xsd”

Once again, has we explain previous, by selecting this option, we will be guided automatically by the tool “BizTalk Flat File Schema Wizard” that will help us to create a Flat File Schema and define its data structure (records, elements, attributes …) based on the text file specified.

• Select “Next” to continue.

• In the window “Flat File Schema Information” we will have to:
  • Select an instance of the text file that will serve as the model of the structure that we want to transform;
  • Although it is not necessary, it is good practice to rename the Record name “Root”.
    • In this case, we will rename it to “Employees”
  • And finally, assign a “Target namespace” to the scheme and define the encoding of the input file

Note: The wizard will load the text file so that we can begin to split it and map it into the desired structure. In this step, we need to define how the records or rows are differentiated.

• Since each record “Employee” that we want to create is defined in a single line, in the “Select Document Data” we can select only the data portion of the document that will set the record, i.e. the whole first line.

Note: Alternatively, let’s call it option 2, we could select all the data of the file, however with this approach we need to treat differently the data a few steps ahead in the “Child Elements” window.

Note: if you notice that some characters are not being parsed correctly as the picture bellow shows, then you probably need to go back to the “Flat File Schema Information” windows and set a different code page, for example, “Western-European (1252)” instead of “UTF-8 (65001)”.

• In the window “Select Record Format”, even though we are dealing with a positional flat file, at this point we are trying to define the structure of each “Employee” record which is delimited by a return Carriage Return/Line Feed ({CR}{LF}). So, in this case, we will have to select the “By delimiter symbol” option.

• In the window “Delimited Record” we will provide the record delimiter, once again, we want to define the structure of Employee, i.e. each row is an Employee node, our limiter is this sample will be the {CR}{LF} (Carriage Return/Line Feed).

• In the window “Child Elements” we will define what kind of element we want to assign to the registry. As we are defining the Employee structure and the file contains multiple employees, we have to select the “Element Type” as “Repeating record” If we do not perform this step, we will not have the ability to split the record into multiple elements / attributes individual.
  • Note: If you do not define the element as “Repeating record” (or “Record”), you will not have the ability to break down your record into individual elements or attributes.

Note: earlier I told that we could select all the data inside the flat file and that we would need to treat the information differently in this step – option 2. So if you select this option, it will present all the lines in the document in this window, because we want to have an “Array” of Employees, in this step we need to select the “Element Type” of the first element as “Repeating record” and the rest of the set as “Ignore” (You set this to Ignore because you have already specified Repeating Record for the Employee node)

In this point, we have defined the translation rules for the repeating record “Employee” but we have not broken down the various elements that compose the record itself. We have just map that each line of the text file corresponds to a record Employee.

• In the “Schema View” select “Next” to continue processing the message

At this point, everything is exactly the same as we did in the Delimited by Symbols example. From here forward is that the changes will emerge.

• In this phase, the wizard restarts the whole process described above, but if you noticed, the wizard no longer selects all information contained in the text file, but only what was selected to define the record Employee. What we will do now is split the information of record “Employee” in different elements, for that, we will select only the information required leaving out the Carriage Return/Line Feed. On the “Select Document Data” screen click “Next”.

• On the “Select Record Format” screen, because now the content of the “Employee” record needs to be parsed according to predefined length (positional), we need to select the option “By relative positions” and then click “Next”

• The wizard provides a visual tool for showing and calculating the distance between the fields. On the “Positional Record” page, Use the left mouse button (or the right) to click a position marker in the position selection box to set the new position marker line. The position marker is represented by a solid line in will represent the rule in which a specific name field begins.
  • Position Marker selection box: Click an area on the page to represent the beginning position of the element. Click again to remove the position marker.
  • By default, a position marker line exists at the beginning of the record at position zero.
  • Click the following position markers to represent the rest of the data fields:

• In the “Child Elements” page, we will define the different elements/attributes of the structure of the record person. This operation is very similar to any XSD, where we can define the different names and data types. Adjust the values to:
  • Have the same name of the table above in the “Element Name”
  • Leave the default “Element Type” and “Date Type” – Field element and String

Note: Once again, you should rename the parent node, each record, element or attribute in the BizTalk Flat File Schema Wizard, mainly in this step by modify the “Element Name” field to your desired name, however if you forgot to do that or if you make a mistake defining one of them, you can always rectify it in the Schema Editor. The same applies to the Data Type.

• Select “Next” to continue.
• Finally, the wizard will show the equivalent XML structure that your text file document will have. Once you select the option “Finish”, the scheme will be available for you to use in your BizTalk solution.

After we finalize the creation of the Flat File Schema which will contain the translation/transformation rules of the text file, we can easily test our parsing rules, without having to get out of our development tool (Visual Studio) and without having to deploy our solution.

If we select the Flat File Schema that we just created and access to its properties, we can see that by default all properties are preconfigured so that we can perform tests to our message transformation:

• The input instance file is configured with the file which was used to create the schema;
• And with the correct input and output formats
  • Validate Instance Input Type: Native
  • Generate Instance Output Type: XML

To test, you simply right-click in the schema that we just created and select the option “Validate Instance”. This option will use the configured file to execute and validate all transformation rules defined in the Flat File Schema and present the final result or occurring errors in the “Output” window

Did you know that Fields in positional records have a limit of 50000000 characters?

Creating the Pipeline that will be responsible for processing and transforming the text file

Once again we need to create at least one custom pipeline (receive pipeline) to process this flat-file document and translate it into is equivalent in XML.

To create the Receive Pipeline that will be responsible for processing and transforming the text file, we need to go to the BizTalk solution created in Visual Studio and perform the following steps:

• Press the right button on top of the project in Solution Explorer, and select the option “Add –> New Item…”.
• On “Installed Templates” menu in the window “Add New Item”, select the option “Pipeline Files”, and then select the option “Receive Pipeline”, then provide the name you want to give the pipeline in this example: “FF_Employee_ReceivePipeline.btp”
• By selecting the option “Add”, the pipeline editor (BizTalk Pipeline Designer) will appear and that will let you view and add components to all the steps associated with receive pipeline: Decode, Disassemble, Validate and ResolveParty.

In this case, the pipeline that we will create it will be responsible for receiving a text file through a receive location and convert it to XML. For that, we will use the “Flat File Disassembler” component which is available in the Visual Studio Toolbox window (Ctrl+W, X).

• Drag it into the Pipeline in step “Disassemble”.

• Finally, select the component “Flat file disassembler”, go to its properties and on the “Document Schema” option: select the schema created earlier, in case the “PositionalTextFiles.FF_Positional_Employees”.

Note: If you want to create a send pipeline in order to transform an XML document into Flat File, we would follow the same steps, the difference is that we would have to drag the component “Flat File Assembler” on stage “Assemble”.

Once you have deployed your solution, you will want to ensure that you select the custom pipeline we just created in the Receive Locations that you expect to receive these flat files.

Now here it comes the same questions: Do you think the Schema is entirely correct? Does it handle all files accordingly to the expected results?

Or do we need to tune some of the properties of your schema… stay tuned to the next post.

Related posts:

• BizTalk Server: Teach me something new about Flat Files (or not) – Introduction
• BizTalk Server: Teach me something new about Flat Files (or not) – Files Delimited by Symbols
• BizTalk Server: Teach me something new about Flat Files (or not) – Delimited Flat File Schema Optimizations

You can download all the code used here from:

BizTalk Server: Dealing with Positional Flat Files (14 KB)
Microsoft | Code Gallery

The post BizTalk Server: Teach me something new about Flat Files (or not) – Positional Files appeared first on SANDRO PEREIRA BIZTALK BLOG.