Skip to content
40 changes: 35 additions & 5 deletions docs/library/CsvProvider.fsx
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,9 @@ The following sample calls the `Load` method with an URL that points to a live C
let msft = Stocks.Load(__SOURCE_DIRECTORY__ + "/../data/MSFT.csv").Cache()

// Look at the most recent row. Note the 'Date' property
// is of type 'DateTime' and 'Open' has a type 'decimal'
// is of type 'DateTime' by default. Set PreferDateOnly = true
// to use 'DateOnly' on .NET 6+
// and 'Open' has a type 'decimal'
let firstRow = msft.Rows |> Seq.head
let lastDate = firstRow.Date
let lastOpen = firstRow.Open
Expand All @@ -107,8 +109,9 @@ collection of rows. We iterate over the rows using a `for` loop. As you can see
to the columns in the CSV file.

As you can see, the type provider also infers types of individual rows. The `Date`
property is inferred to be a `DateTime` (because the values in the sample file can all
be parsed as dates) while HLOC prices are inferred as `decimal`.
property is inferred as `DateTime` by default. When you set `PreferDateOnly = true`
on .NET 6 and later, date-only strings (without a time component) are inferred as `DateOnly`.
HLOC prices are inferred as `decimal`.

## Using units of measure

Expand Down Expand Up @@ -269,8 +272,12 @@ it by specifying the `InferRows` static parameter of `CsvProvider`. If you speci
Columns with only `0`, `1`, `Yes`, `No`, `True`, or `False` will be set to `bool`. Columns with numerical values
will be set to either `int`, `int64`, `decimal`, or `float`, in that order of preference.

If a value is missing in any row, by default the CSV type provider will infer a nullable (for `int` and `int64`) or an optional
(for `bool`, `DateTime` and `Guid`). When a `decimal` would be inferred but there are missing values, we will infer a
On .NET 6 and later, when you set `PreferDateOnly = true`, columns whose values are all date-only strings (without a time component, e.g. `2023-01-15`)
are inferred as `DateOnly`. Time-only strings are inferred as `TimeOnly`. If a column mixes `DateOnly` and `DateTime` values, it is unified to `DateTime`.
By default (`PreferDateOnly = false`), all date values are inferred as `DateTime` for backward compatibility.

If a value is missing in any row, by default the CSV type provider will infer a nullable (for `int`, `int64`, and `DateOnly`) or an optional
(for `bool`, `DateTime`, `DateTimeOffset`, and `Guid`). When a `decimal` would be inferred but there are missing values, we will infer a
`float` instead, and use `Double.NaN` to represent those missing values. The `string` type is already inherently nullable,
so by default, we won't generate a `string option`. If you prefer to use optionals in all cases, you can set the static parameter
`PreferOptionals` to `true`. In that case, you'll never get an empty string or a `Double.NaN` and will always get a `None` instead.
Expand Down Expand Up @@ -303,6 +310,12 @@ specify the units of measure. This will override both `AssumeMissingValues` and
* `guid`
* `guid?`
* `guid option`
* `dateonly` (.NET 6+ only)
* `dateonly?` (.NET 6+ only)
* `dateonly option` (.NET 6+ only)
* `timeonly` (.NET 6+ only)
* `timeonly?` (.NET 6+ only)
* `timeonly option` (.NET 6+ only)
* `string`
* `string option`.

Expand Down Expand Up @@ -373,6 +386,23 @@ for row in titanic2.Rows |> Seq.truncate 10 do

You can even mix and match the two syntaxes like this `Schema="int64,DidSurvive,PClass->Passenger Class=string"`

### DateOnly and TimeOnly (on .NET 6+)

On .NET 6 and later, when you set `PreferDateOnly = true`, date-only strings are inferred as `DateOnly`
and time-only strings as `TimeOnly`. For example, a column like `EventDate` containing values such as
`2023-06-01` will be given the type `DateOnly`. By default (`PreferDateOnly = false`), dates are
inferred as `DateTime` for backward compatibility.

You can also explicitly request a `DateOnly` or `TimeOnly` column using schema annotations:

[lang=text]
EventDate (dateonly),Duration (timeonly?)
2023-06-01,08:30:00
2023-06-02,

In the example above, `EventDate` is explicitly annotated as `dateonly` and `Duration` is explicitly
annotated as `timeonly?` (a nullable `TimeOnly`).

## Transforming CSV files

In addition to reading, `CsvProvider` also has support for transforming the row collection of CSV files. The operations
Expand Down
7 changes: 7 additions & 0 deletions docs/library/JsonProvider.fsx
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,13 @@ type-safe access to the values, but not in the original order (if order matters,
you can use the `mixed.JsonValue` property to get the underlying `JsonValue` and
process it dynamically as described in [the documentation for `JsonValue`](JsonValue.html).

### Inferring date types

String values in JSON that look like dates are inferred as `DateTime` or `DateTimeOffset`.
On .NET 6 and later, when you set `PreferDateOnly = true`, strings that represent a date without a time component (e.g. `"2023-01-15"`)
are inferred as `DateOnly`, and time-only strings are inferred as `TimeOnly`. By default (`PreferDateOnly = false`),
all dates are inferred as `DateTime` for backward compatibility.

### Inferring record types

Now let's look at a sample JSON document that contains a list of records. The
Expand Down
9 changes: 8 additions & 1 deletion src/FSharp.Data.Csv.Core/CsvInference.fs
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,14 @@ let private nameToTypeForCsv =
"datetimeoffset option", (typeof<DateTimeOffset>, TypeWrapper.Option)
"timespan option", (typeof<TimeSpan>, TypeWrapper.Option)
"guid option", (typeof<Guid>, TypeWrapper.Option)
"string option", (typeof<string>, TypeWrapper.Option) ]
"string option", (typeof<string>, TypeWrapper.Option)
#if NET6_0_OR_GREATER
"dateonly?", (typeof<DateOnly>, TypeWrapper.Nullable)
"timeonly?", (typeof<TimeOnly>, TypeWrapper.Nullable)
"dateonly option", (typeof<DateOnly>, TypeWrapper.Option)
"timeonly option", (typeof<TimeOnly>, TypeWrapper.Option)
#endif
]
|> dict

let private nameAndTypeRegex =
Expand Down
2 changes: 1 addition & 1 deletion src/FSharp.Data.Csv.Core/FSharp.Data.Csv.Core.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Library</OutputType>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFrameworks>netstandard2.0;net8.0</TargetFrameworks>
<OtherFlags>$(OtherFlags) --warnon:1182 --nowarn:10001 --nowarn:44</OtherFlags>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,12 @@ let getConversionQuotation missingValuesStr cultureStr typ (value: Expr<string o
<@@ TextRuntime.ConvertDateTimeOffset(cultureStr, %value) @@>
elif typ = typeof<TimeSpan> then
<@@ TextRuntime.ConvertTimeSpan(cultureStr, %value) @@>
#if NET6_0_OR_GREATER
elif typ = typeof<DateOnly> then
<@@ TextRuntime.ConvertDateOnly(cultureStr, %value) @@>
elif typ = typeof<TimeOnly> then
<@@ TextRuntime.ConvertTimeOnly(cultureStr, %value) @@>
#endif
elif typ = typeof<Guid> then
<@@ TextRuntime.ConvertGuid(%value) @@>
else
Expand All @@ -58,6 +64,12 @@ let getBackConversionQuotation missingValuesStr cultureStr typ value : Expr<stri
<@ TextRuntime.ConvertDateTimeOffsetBack(cultureStr, %%value) @>
elif typ = typeof<TimeSpan> then
<@ TextRuntime.ConvertTimeSpanBack(cultureStr, %%value) @>
#if NET6_0_OR_GREATER
elif typ = typeof<DateOnly> then
<@ TextRuntime.ConvertDateOnlyBack(cultureStr, %%value) @>
elif typ = typeof<TimeOnly> then
<@ TextRuntime.ConvertTimeOnlyBack(cultureStr, %%value) @>
#endif
else
failwith "getBackConversionQuotation: Unsupported primitive type"

Expand Down
11 changes: 11 additions & 0 deletions src/FSharp.Data.DesignTime/CommonProviderImplementation/Helpers.fs
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,17 @@ module internal ProviderHelpers =
member x.Inverse(denominator) : Type =
ProvidedMeasureBuilder.Inverse(denominator) }

#if NET6_0_OR_GREATER
/// Returns true when the target runtime assembly is a .NET 6+ build and therefore
/// supports System.DateOnly / System.TimeOnly in generated types.
let runtimeSupportsNet6Types (runtimeAssemblyPath: string) =
// The assembly path contains the TFM, e.g. "…/net8.0/FSharp.Data.dll".
// Anything matching "/net<N>." where N β‰₯ 6 is a net6+ target.
let path = runtimeAssemblyPath.Replace('\\', '/').ToLowerInvariant()
let m = System.Text.RegularExpressions.Regex.Match(path, @"/net(\d+)\.")
m.Success && (int m.Groups.[1].Value) >= 6
#endif

let asyncMap (resultType: Type) (valueAsync: Expr<Async<'T>>) (body: Expr<'T> -> Expr) =
let (?) = QuotationBuilder.(?)
let convFunc = ReflectionHelpers.makeDelegate (Expr.Cast >> body) typeof<'T>
Expand Down
36 changes: 24 additions & 12 deletions src/FSharp.Data.DesignTime/Csv/CsvProvider.fs
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ type public CsvProvider(cfg: TypeProviderConfig) as this =
let encodingStr = args.[13] :?> string
let resolutionFolder = args.[14] :?> string
let resource = args.[15] :?> string
let preferDateOnly = args.[16] :?> bool

// This provider already has a schema mechanism, so let's disable inline schemas.
let inferenceMode = InferenceMode'.ValuesOnly
Expand Down Expand Up @@ -103,16 +104,25 @@ type public CsvProvider(cfg: TypeProviderConfig) as this =
let inferredFields =
use _holder = IO.logTime "Inference" sample

sampleCsv.InferColumnTypes(
inferRows,
TextRuntime.GetMissingValues missingValuesStr,
inferenceMode,
TextRuntime.GetCulture cultureStr,
schema,
assumeMissingValues,
preferOptionals,
unitsOfMeasureProvider
)
let fields =
sampleCsv.InferColumnTypes(
inferRows,
TextRuntime.GetMissingValues missingValuesStr,
inferenceMode,
TextRuntime.GetCulture cultureStr,
schema,
assumeMissingValues,
preferOptionals,
unitsOfMeasureProvider
)
#if NET6_0_OR_GREATER
if preferDateOnly && ProviderHelpers.runtimeSupportsNet6Types cfg.RuntimeAssembly then
fields
else
fields |> List.map StructuralInference.downgradeNet6PrimitiveProperty
#else
fields
#endif

use _holder = IO.logTime "TypeGeneration" sample

Expand Down Expand Up @@ -223,7 +233,8 @@ type public CsvProvider(cfg: TypeProviderConfig) as this =
ProvidedStaticParameter("Culture", typeof<string>, parameterDefaultValue = "")
ProvidedStaticParameter("Encoding", typeof<string>, parameterDefaultValue = "")
ProvidedStaticParameter("ResolutionFolder", typeof<string>, parameterDefaultValue = "")
ProvidedStaticParameter("EmbeddedResource", typeof<string>, parameterDefaultValue = "") ]
ProvidedStaticParameter("EmbeddedResource", typeof<string>, parameterDefaultValue = "")
ProvidedStaticParameter("PreferDateOnly", typeof<bool>, parameterDefaultValue = false) ]

let helpText =
"""<summary>Typed representation of a CSV file.</summary>
Expand All @@ -246,7 +257,8 @@ type public CsvProvider(cfg: TypeProviderConfig) as this =
<param name='Encoding'>The encoding used to read the sample. You can specify either the character set name or the codepage number. Defaults to UTF8 for files, and to ISO-8859-1 the for HTTP requests, unless <c>charset</c> is specified in the <c>Content-Type</c> response header.</param>
<param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution).</param>
<param name='EmbeddedResource'>When specified, the type provider first attempts to load the sample from the specified resource
(e.g. 'MyCompany.MyAssembly, resource_name.csv'). This is useful when exposing types generated by the type provider.</param>"""
(e.g. 'MyCompany.MyAssembly, resource_name.csv'). This is useful when exposing types generated by the type provider.</param>
<param name='PreferDateOnly'>When true on .NET 6+, date-only strings are inferred as DateOnly and time-only strings as TimeOnly. Defaults to false for backward compatibility.</param>"""

do csvProvTy.AddXmlDoc helpText
do csvProvTy.DefineStaticParameters(parameters, buildTypes)
Expand Down
2 changes: 1 addition & 1 deletion src/FSharp.Data.DesignTime/FSharp.Data.DesignTime.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<IsPackable>false</IsPackable>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFrameworks>netstandard2.0;net8.0</TargetFrameworks>
<DefineConstants>IS_DESIGNTIME;NO_GENERATIVE;$(DefineConstants)</DefineConstants>
<OtherFlags>$(OtherFlags) --warnon:1182 --nowarn:44</OtherFlags>
<GenerateDocumentationFile>false</GenerateDocumentationFile>
Expand Down
59 changes: 51 additions & 8 deletions src/FSharp.Data.DesignTime/Html/HtmlGenerator.fs
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,11 @@ module internal HtmlGenerator =
let private createTableType
getTableTypeName
(inferenceParameters, missingValuesStr, cultureStr)
supportsNet6Types
(table: HtmlTable)
=

let columns =
let rawColumns =
match table.InferedProperties with
| Some inferedProperties -> inferedProperties
| None ->
Expand All @@ -52,6 +53,16 @@ module internal HtmlGenerator =
else
table.Rows)

let columns =
#if NET6_0_OR_GREATER
if supportsNet6Types then
rawColumns
else
rawColumns |> List.map StructuralInference.downgradeNet6PrimitiveProperty
#else
rawColumns
#endif

let fields =
columns
|> List.mapi (fun index field ->
Expand Down Expand Up @@ -128,9 +139,24 @@ module internal HtmlGenerator =

create, tableType

let private createListType getListTypeName (inferenceParameters, missingValuesStr, cultureStr) (list: HtmlList) =
let private createListType
getListTypeName
(inferenceParameters, missingValuesStr, cultureStr)
supportsNet6Types
(list: HtmlList)
=

let rawColumns = HtmlInference.inferListType inferenceParameters list.Values

let columns = HtmlInference.inferListType inferenceParameters list.Values
let columns =
#if NET6_0_OR_GREATER
if supportsNet6Types then
rawColumns
else
StructuralInference.downgradeNet6Types rawColumns
#else
rawColumns
#endif

let listItemType, conv =
match columns with
Expand Down Expand Up @@ -185,14 +211,25 @@ module internal HtmlGenerator =
let private createDefinitionListType
getDefinitionListTypeName
(inferenceParameters, missingValuesStr, cultureStr)
supportsNet6Types
(definitionList: HtmlDefinitionList)
=

let getListTypeName = typeNameGenerator ()

let createListType index (list: HtmlList) =

let columns = HtmlInference.inferListType inferenceParameters list.Values
let rawColumns = HtmlInference.inferListType inferenceParameters list.Values

let columns =
#if NET6_0_OR_GREATER
if supportsNet6Types then
rawColumns
else
StructuralInference.downgradeNet6Types rawColumns
#else
rawColumns
#endif

let listItemType, conv =
match columns with
Expand Down Expand Up @@ -264,7 +301,7 @@ module internal HtmlGenerator =

definitionListType

let generateTypes asm ns typeName parameters htmlObjects =
let generateTypes asm ns typeName parameters supportsNet6Types htmlObjects =

let htmlType =
ProvidedTypeDefinition(
Expand Down Expand Up @@ -303,7 +340,10 @@ module internal HtmlGenerator =
match htmlObj with
| Table table ->
let containerType = getOrCreateContainer "Tables"
let create, tableType = createTableType getTypeName parameters table

let create, tableType =
createTableType getTypeName parameters supportsNet6Types table

htmlType.AddMember tableType

containerType.AddMember
Expand All @@ -315,7 +355,7 @@ module internal HtmlGenerator =

| List list ->
let containerType = getOrCreateContainer "Lists"
let create, tableType = createListType getTypeName parameters list
let create, tableType = createListType getTypeName parameters supportsNet6Types list
htmlType.AddMember tableType

containerType.AddMember
Expand All @@ -326,7 +366,10 @@ module internal HtmlGenerator =
)
| DefinitionList definitionList ->
let containerType = getOrCreateContainer "DefinitionLists"
let tableType = createDefinitionListType getTypeName parameters definitionList

let tableType =
createDefinitionListType getTypeName parameters supportsNet6Types definitionList

htmlType.AddMember tableType

containerType.AddMember
Expand Down
Loading
Loading