`ConvertFrom-Yaml` doesn't enumerate top-level sequences in pipeline output

[skaravos]

Summary

When ConvertFrom-Yaml parses a top-level YAML sequence (array), the result is written to the pipeline as a single Generic.List1[Object] rather than unrolling into individual elements. This breaks direct piping to downstream cmdlets like Select-Object or Where-Object.

Expected Behaviour

# enumerable (top-level) input sequences should unroll like ConvertFrom-Json
> '[1,2,3]' | ConvertFrom-Json | Select-Object -Last 1 
3

# enumerable (top-level) input sequences should unroll like ConvertFrom-Json
> '[1,2,3]' | ConvertFrom-Yaml | Select-Object -Last 1 
3

Actual Behaviour

# enumerable input sequences don't unroll, they are passed down the pipeline as one list object
> '[1,2,3]' | ConvertFrom-Yaml | Select-Object -Last 1
1
2
3

Workarounds

NOTE: this appears to be the "intended" use of the module as all examples in the README store output into an intermediate variable

# storing output into a variable lets PowerShell unroll the list
> $x = '[1,2,3]' | ConvertFrom-Yaml
> $x | Select-Object -Last 1
3

# collecting output with parentheses also works
> ('[1,2,3]' | ConvertFrom-Yaml) | Select-Object -Last 1 
3

# inject an empty trailing YAML document (I think this difference in behavior is unintended)
> "[1,2,3]`n---" | ConvertFrom-Yaml -AllDocuments | Select-Object -Last 1 
3

Suspected cause

I think the issue lies on powershell-yaml:493

# powershell-yaml.psm1

function Convert-YamlSequenceToArray {
    # ...
    $ret = [System.Collections.Generic.List[object]](New-Object 'System.Collections.Generic.List[object]')
    # ...
    return , $ret #<-- explicitly prevents unrolling (I think to preserve nested lists??)
    # ...
}
# ...
function Convert-YamlDocumentToPSObject {
    # ...
    'YamlDotNet.RepresentationModel.YamlSequenceNode' {
        return Convert-YamlSequenceToArray $Node -Ordered:$Ordered
    }
    # ...
}
# ...
function ConvertFrom-Yaml {
    # ...
    if (($documents.Count -eq 1) -or !$AllDocuments) {
        return Convert-YamlDocumentToPSObject $documents[0].RootNode -Ordered:$Ordered  #<-- directly returns List[Object] without any unrolling, even for the top-level sequence
    }
    # with more than one document and the `-AllDocuments` switch, it assigns to an intermediate variable, so it does unroll the lists
    $ret = @()
    foreach ($i in $documents) {
        $ret += Convert-YamlDocumentToPSObject $i.RootNode -Ordered:$Ordered
    }
    return $ret
    # ...
}

Possible fix?

Enumerate the list/array results at the ConvertFrom-Yaml cmdlet boundary by either explicitly iterating, by adding parentheses around the Convert-YamlDocumentToPSObject call, or by storing the list into an intermediate variable (which allows powershell to implicitly unroll it).

function ConvertFrom-Yaml {
 # ...
  if (($documents.Count -eq 1) -or !$AllDocuments) {
     $result = Convert-YamlDocumentToPSObject $documents[0].RootNode -Ordered:$Ordered #<-- implicitly unrolls top-level sequence
     return $result
  }
  # ...
}

I applied the above change on my machine and it seems to have fixed the pipeline output for the few YAML arrays I was working with, but I obviously haven't tested all the other varieties of YAML documents, where I'm sure there are edge-cases I'm overlooking.

# with the above change to ConvertFrom-Yaml applied, arrays can be iterated in the pipeline:
> '[1,2,3]' | ConvertFrom-Yaml | Select-Object -Last 1 
3

# and nested arrays also appear to work
> '[[9,8,7],[6,5,4],[3,2,1]]' | ConvertFrom-Yaml | Select-Object -Last 1 
3
2
1

[gabriel-samfira]

Hi @skaravos

The reason we prevent unrolling at line #493 is because of a powershell behavior that casts single element arrays to the type of the single element within the array:

function Get-Array {
    return @(1)
}

$x = Get-Array
$x.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int32                                    System.ValueType

Which when deserializing a yaml with an array that has a single element will turn:

anImportantArray:
  - 1

to

anImportantArray: 1

Which breaks all sorts of things.

I will try to allocate some time to fix this.

[skaravos]

I agree that the prevention of unrolling inside the internal Convert-YamlSequenceToArray function is extremely important for preserving nested arrays and single-element arrays, this should definitely not be changed.

My thoughts are that the unrolling behavior for arrays matters less at the "top-level", especially where it breaks the "normal" PowerShell pipeline behavior.

Just as an example, even if you were to use an intermediate variable in the ConvertFrom-Yaml function as I suggested as the "Possible Fix", the code you show would still perform exactly the same because the array is the value of anImportantArray, it isn't itself the top-level sequence of the document.

# even with the change to unroll top-level sequences, there is no difference in behavior here:
> 'anImportantArray: [1]' | ConvertFrom-Yaml

Name                           Value
----                           -----
anImportantArray               {1}

It would only cause differences for a top-level sequence with one element:

# as it is right now, the sequence is passed as a single-element List
> ('[1]' | ConvertFrom-Yaml).GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     List`1                                   System.Object


# with the suggested top-level unrolling change, it would be unrolled
> ('[1]' | ConvertFrom-Yaml).GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int32                                    System.ValueType

It is worth noting that this unrolling already happens when using the -AllDocuments flag, where each separate top-level sequence is unrolled, it is just masked by explicitly re-wrapping everything in a plain System.Array[] afterwards.

# this is the current behavior of -AllDocuments:
> $yaml = @"
- 1
---
- 2
- 3
"@
> $yaml | ConvertFrom-Yaml -AllDocuments |  % { $_.GetType() }

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int32                                    System.ValueType
True     True     Int32                                    System.ValueType
True     True     Int32                                    System.ValueType

# if it were symmetrical with single-document parsing, we would instead expect to see this:
> $yaml = @"
- 1
---
- 2
- 3
"@
> $yaml | ConvertFrom-Yaml -AllDocuments |  % { $_.GetType() }

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     List`1                                   System.Object
True     True     List`1                                   System.Object

[skaravos]

The problem I'm describing has already been debated for the built-in ConvertFrom-Json cmdlet: PowerShell/PowerShell#3424