Skip to content

Commit

Permalink
[Infrastructure] Move back to NPM (dotnet#52914)
Browse files Browse the repository at this point in the history
This change removes Yarn from our infrastructure completely and replaces it with NPM. There is a package.json in the top folder of the respository that defines the workspace. The workspace lists all the paths to packages that are part of the repository. In addition to that, the top package.json defines scripts that can be run from the root of the repository to install, build test, etc. all the packages in the repository.

Each package has its own package.json file that defines its dependencies and scripts. The scripts in the top package.json file are just wrappers around the scripts in the individual package.json files.

## Developer workflow

### Restore dependencies

To install the dependencies, you can run npm ci from the root of the repository. This will install all the dependencies for all the packages in the repository and automatically link them together. This normally takes very little time once you've have downloaded the packages onto your machine (10s for me).

### Install or update new dependencies
Run npm install ... on whatever node you want to install the dependencies. The package-lock.json will get updated accordingly. For installing new dependencies you might need to authenticate yourself with the NPM VSTS feed. To do that, run:

* `npm install -g vsts-npm-auth`
* `vsts-npm-auth -config .npmrc -F`. This will walk you through the auth process and provision the required credentials.
*  `npm install`

### Updating deps to fix Component Governance alerts

* Run npm audit to see the list of issues.
* Run npm audit fix to see if the issues can get addressed automatically.
* If that's not the case:
  * If possible, update to a newer version of the package that doesn't include the vulnerable dependency.
  * If that's not possible, at the appropriate node, add an entry to the "overrides" property to force the transitive dependency into a non-vulnerable version.
  * run npm install from the root.
  * run npm audit again to see if the issues have disappeared, if they haven't continue making changes (update more packages, etc).

Upgrading the main deps should always be the preferred route over using overrides. Overrides require that we periodically remove them to check if we can update the dependency later on.

### Building JS components

You can run `npm run build` at the repo root and it will build all the projects (takes about 2 minutes). Or alternatively, you can run `npm run build` on an individual project.json to build only that `project.json`. If you do this, any dependent project should have been built previously. Dependencies won't get build automatically.

### Testing JS components

There are two test categories:
 * Unit tests: They are self-contained and don't depend on any of the .NET Code. Run `npm run test`.
 * Integration tests: They test the JS in combination with the .NET Code, for which you first need to have run `eng\Build.cmd`, before you invoke the tests. To execute them, run `npm run integration-test`
  • Loading branch information
javiercn authored Dec 22, 2023
1 parent 85e457c commit 8c27269
Show file tree
Hide file tree
Showing 93 changed files with 20,049 additions and 23,633 deletions.
4 changes: 2 additions & 2 deletions .azure/pipelines/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -750,9 +750,9 @@ stages:
binlogPath: artifacts/log/Release/Build.binlog
presteps:
- task: NodeTool@0
displayName: Install Node 18.x
displayName: Install Node 20.x
inputs:
versionSpec: 18.x
versionSpec: 20.x
pool:
name: $(DncEngInternalBuildPool)
demands: ImageOverride -equals 1es-windows-2022
Expand Down
4 changes: 3 additions & 1 deletion .azure/pipelines/components-e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@ jobs:
displayName: Update submodules
- script: ./restore.sh
displayName: Run restore.sh
- script: yarn install --frozen-lockfile --cwd ./src/Components/test/E2ETest || yarn install --frozen-lockfile --cwd ./src/Components/test/E2ETest
- script: npm ci
displayName: NPM install
- script: npm run build
displayName: Build JS
- script: .dotnet/dotnet build ./src/Components/test/E2ETest -c $(BuildConfiguration) --no-restore
displayName: Build
- script: .dotnet/dotnet test ./src/Components/test/E2ETest -c $(BuildConfiguration) --no-build --filter 'Quarantined!=true|Quarantined=false'
Expand Down
4 changes: 2 additions & 2 deletions .azure/pipelines/jobs/default-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -199,9 +199,9 @@ jobs:
displayName: Start background dump collection
- ${{ if eq(parameters.installNodeJs, 'true') }}:
- task: NodeTool@0
displayName: Install Node 18.x
displayName: Install Node 20.x
inputs:
versionSpec: 18.x
versionSpec: 20.x
- ${{ if and(eq(parameters.installJdk, 'true'), eq(parameters.agentOs, 'Windows')) }}:
- powershell: ./eng/scripts/InstallJdk.ps1
displayName: Install JDK 11
Expand Down
1 change: 0 additions & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,6 @@
<Import Project="eng\targets\Cpp.Common.props" Condition="'$(MSBuildProjectExtension)' == '.vcxproj'" />
<Import Project="eng\targets\CSharp.Common.props" Condition="'$(MSBuildProjectExtension)' == '.csproj'" />
<Import Project="eng\targets\Wix.Common.props" Condition="'$(MSBuildProjectExtension)' == '.wixproj'" />
<Import Project="eng\targets\Npm.Common.props" Condition="'$(MSBuildProjectExtension)' == '.npmproj'" />
<Import Project="eng\targets\Java.Common.props" Condition="'$(MSBuildProjectExtension)' == '.javaproj'" />
<Import Project="eng\testing\linker\trimmingTests.props" Condition="'$(IsPublishedAppTestProject)' == 'true'" />
<Import Project="eng\targets\Helix.props" Condition=" $(IsTestProject) " />
Expand Down
1 change: 0 additions & 1 deletion Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,6 @@
<Import Project="eng\targets\CSharp.Common.targets" Condition="'$(MSBuildProjectExtension)' == '.csproj'" />
<Import Project="eng\targets\FSharp.Common.targets" Condition="'$(MSBuildProjectExtension)' == '.fsproj'" />
<Import Project="eng\targets\Wix.Common.targets" Condition="'$(MSBuildProjectExtension)' == '.wixproj'" />
<Import Project="eng\targets\Npm.Common.targets" Condition="'$(MSBuildProjectExtension)' == '.npmproj'" />
<Import Project="eng\targets\Java.Common.targets" Condition="'$(MSBuildProjectExtension)' == '.javaproj'" />
<Import Project="eng\testing\linker\trimmingTests.targets" Condition="'$(IsPublishedAppTestProject)' == 'true'" />
<Import Project="eng\targets\Helix.targets" Condition=" $(IsTestProject) " />
Expand Down
23 changes: 9 additions & 14 deletions eng/Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@
<ItemGroup>
<ProjectToBuild Include="$(ProjectToBuild)"
Exclude="@(ProjectToExclude);$(RepoRoot)**\bin\**\*;$(RepoRoot)**\obj\**\*">
<BuildInParallel Condition=" '%(Extension)' == '.npmproj' OR '%(Extension)' == '.vcxproj' ">false</BuildInParallel>
<RestoreInParallel Condition=" '%(Extension)' == '.npmproj' ">false</RestoreInParallel>
<BuildInParallel Condition=" '%(Extension)' == '.nodeproj' OR '%(Extension)' == '.vcxproj' ">false</BuildInParallel>
<RestoreInParallel Condition=" '%(Extension)' == '.nodeproj' ">false</RestoreInParallel>
<!-- Also do not build in parallel w/in npm projects. -->
<AdditionalProperties Condition=" '%(Extension)' == '.npmproj' OR '%(Extension)' == '.vcxproj' ">BuildInParallel=false</AdditionalProperties>
<AdditionalProperties Condition=" '%(Extension)' == '.nodeproj' OR '%(Extension)' == '.vcxproj' ">BuildInParallel=false</AdditionalProperties>
</ProjectToBuild>
</ItemGroup>
</When>
Expand Down Expand Up @@ -124,25 +124,19 @@
<ProjectToBuild Condition=" $(BuildNative) " Include="@(NativeProjects)" Exclude="@(ProjectToExclude)" />
<ProjectToExclude Condition=" !$(BuildNative) " Include="@(NativeProjects)" />

<NodeJsProjects Include="
$(RepoRoot)src\Components\Web.JS\Microsoft.AspNetCore.Components.Web.JS.npmproj;
$(RepoRoot)src\SignalR\**\*.npmproj;
$(RepoRoot)src\JSInterop\**\*.npmproj;
"
<NodeJsProjects Condition="'$(TargetArchitecture)' == 'x64'"
Include="$(RepoRoot)eng\Npm.Workspace.nodeproj;
$(RepoRoot)eng\Npm.Workspace.FunctionalTests.nodeproj;"
AdditionalProperties="BuildInParallel=false"
BuildInParallel="false"
RestoreInParallel="false"
Exclude="@(ProjectToExclude)" />

<ExplicitRequiresDelay Include="$(RepoRoot)eng\Npm.Workspace.FunctionalTests.nodeproj" />

<ProjectToBuild Condition=" '$(BuildNodeJS)' == 'true'" Include="@(NodeJsProjects)" Exclude="@(ProjectToExclude)" />
<ProjectToExclude Condition=" '$(BuildNodeJS)' != 'true'" Include="@(NodeJsProjects)" />

<YarnMSBuildProjects Include="
$(RepoRoot)src\Components\test\E2ETest\Microsoft.AspNetCore.Components.E2ETests.csproj;
$(RepoRoot)src\Components\WebAssembly\Authentication.Msal\src\Microsoft.Authentication.WebAssembly.Msal.csproj;
$(RepoRoot)src\Components\WebAssembly\WebAssembly.Authentication\src\Microsoft.AspNetCore.Components.WebAssembly.Authentication.csproj;
"
Exclude="@(ProjectToExclude)" />
<ProjectToExclude Condition=" '$(DotNetBuildFromSource)' == 'true'" Include="@(YarnMSBuildProjects)" />

<JavaProjects Include="$(RepoRoot)src\SignalR\**\*.javaproj"
Expand Down Expand Up @@ -253,4 +247,5 @@
</ItemGroup>
</Otherwise>
</Choose>

</Project>
2 changes: 1 addition & 1 deletion eng/CodeGen.proj
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<Target Name="GenerateProjectList">
<Message Importance="High" Text="Analyzing @(ProjectToBuild->Count()) projects" />

<MSBuild Projects="@(ProjectToBuild)"
<MSBuild Projects="@(ProjectToBuild);@(ExplicitRequiresDelay)"
Targets="GetReferencesProvided"
BuildInParallel="true"
SkipNonexistentTargets="true"
Expand Down
53 changes: 53 additions & 0 deletions eng/Npm.Workspace.FunctionalTests.nodeproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<Project>

<!-- Import Directory.Build.Props -->
<Import Project="$(MSBuildThisFileDirectory)..\Directory.Build.props" />

<PropertyGroup>
<IsTestProject>true</IsTestProject>
<IsUnitTestProject>true</IsUnitTestProject>
<IsIntegrationTestProject>true</IsIntegrationTestProject>
<TestDependsOnAspNetPackages>false</TestDependsOnAspNetPackages>
<TestDependsOnAspNetAppPackages>false</TestDependsOnAspNetAppPackages>
<TestDependsOnAspNetRuntime>false</TestDependsOnAspNetRuntime>
<ContainsFunctionalTestAssets>false</ContainsFunctionalTestAssets>
<BuildHelixPayload>false</BuildHelixPayload>
<PackageVersion>$(VersionPrefix)$(VersionSuffix)</PackageVersion>
<RequiresDelayedBuild>true</RequiresDelayedBuild>
<TestGroupName>FunctionalTests</TestGroupName>
<RuntimeIdentifier Condition="'$(TargetFrameworkIdentifier)' == '.NETFramework'">win-x86</RuntimeIdentifier>
</PropertyGroup>

<Target Name="Restore" />
<Target Name="Build" />
<Target Name="Pack" />
<Target Name="Publish" />

<Target Name="_Test" BeforeTargets="Test">
<Message Text="Testing NPM packages..." Importance="high" />
<Exec Command="npm run integration-test" WorkingDirectory="$(MSBuildThisFileDirectory).." />
</Target>

<Target Name="GetReferencesProvided" Returns="@(ProvidesReference)">
<ItemGroup>
<!-- Include File name without the extension -->
<ProvidesReference Include="Npm.Workspace.FunctionalTests">
<IsAspNetCoreApp>$([MSBuild]::ValueOrDefault($(IsAspNetCoreApp),'false'))</IsAspNetCoreApp>
<IsPackable>$([MSBuild]::ValueOrDefault($(IsPackable),'false'))</IsPackable>
<ProjectFileRelativePath>$([MSBuild]::MakeRelative($(RepoRoot), $(MSBuildProjectFullPath)))</ProjectFileRelativePath>
<IsTrimmable>$([MSBuild]::ValueOrDefault($(IsTrimmable),'false'))</IsTrimmable>
<IsShippingPackage>$([MSBuild]::ValueOrDefault($(IsShippingPackage),'false'))</IsShippingPackage>

<!-- True if the project may be referenced using a @(Reference) item. -->
<IsProjectReferenceProvider>$([MSBuild]::ValueOrDefault($(IsProjectReferenceProvider),'false'))</IsProjectReferenceProvider>

<!-- True if project must be restored etc. after App.Ref and App.Runtime are fully built. -->
<RequiresDelayedBuild>$([MSBuild]::ValueOrDefault($(RequiresDelayedBuild),'false'))</RequiresDelayedBuild>
</ProvidesReference>
</ItemGroup>
</Target>

<!-- Import Directory.Build.targets -->
<Import Project="$(MSBuildThisFileDirectory)..\Directory.Build.targets" />

</Project>
41 changes: 41 additions & 0 deletions eng/Npm.Workspace.nodeproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<Project>

<!-- Import Directory.Build.Props -->
<Import Project="$(MSBuildThisFileDirectory)..\Directory.Build.props" />

<PropertyGroup>
<IsTestProject>true</IsTestProject>
<IsUnitTestProject>true</IsUnitTestProject>
<TestDependsOnAspNetPackages>false</TestDependsOnAspNetPackages>
<TestDependsOnAspNetAppPackages>false</TestDependsOnAspNetAppPackages>
<TestDependsOnAspNetRuntime>false</TestDependsOnAspNetRuntime>
<ContainsFunctionalTestAssets>false</ContainsFunctionalTestAssets>
<BuildHelixPayload>false</BuildHelixPayload>
<PackageVersion>$(VersionPrefix)$(VersionSuffix)</PackageVersion>
</PropertyGroup>

<Target Name="Restore">
<Message Text="Restoring NPM packages..." Importance="high" />
<Exec Command="npm ci" WorkingDirectory="$(MSBuildThisFileDirectory).." />
</Target>

<Target Name="Build">
<Message Text="Building NPM packages..." Importance="high" />
<Exec Command="npm run build" WorkingDirectory="$(MSBuildThisFileDirectory).." />
</Target>

<Target Name="_Test" BeforeTargets="Test">
<Message Text="Testing NPM packages..." Importance="high" />
<Exec Command="npm run test" ContinueOnError="true" WorkingDirectory="$(MSBuildThisFileDirectory).." />
</Target>

<Target Name="Pack">
<Message Text="Packing NPM packages..." Importance="high" />
<MakeDir Directories="$(PackageOutputPath)" Condition="!Exists('$(PackageOutputPath)')" />
<Exec Command="node $(MSBuildThisFileDirectory)scripts\pack-workspace.js $(RepoRoot)package.json $(PackageVersion) $(PackageOutputPath)" />
</Target>

<!-- Import Directory.Build.targets -->
<Import Project="$(MSBuildThisFileDirectory)..\Directory.Build.targets" />

</Project>
8 changes: 4 additions & 4 deletions eng/Publishing.props
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@
DependsOnTargets="_WriteProductVersionFile">
<!--
This target is defined in eng/targets/Packaging.targets and Npm.Common.targets and included in every C#, F#,
and npm project. We use SignalR.Npm.FunctionalTests.npmproj because it is non-shipping (we need a non-stable
and npm project. We use SignalR.Npm.FunctionalTests.nodeproj because it is non-shipping (we need a non-stable
version string to use as our publish location), non-packed (won't be shipped in the future), and it is _not_ a
C# or F# project. For now at least, C# and F# projects should not be referenced when using desktop msbuild.
-->
<MSBuild Projects="$(RepoRoot)src\SignalR\clients\ts\FunctionalTests\SignalR.Npm.FunctionalTests.npmproj"
<MSBuild Projects="$(RepoRoot)src\SignalR\clients\ts\FunctionalTests\SignalR.Npm.FunctionalTests.nodeproj"
Properties="DisableYarnCheck=true;ExcludeFromBuild=false"
Targets="_GetPackageVersionInfo">
<Output TaskParameter="TargetOutputs" ItemName="_ResolvedPackageVersionInfo" />
Expand Down Expand Up @@ -102,11 +102,11 @@
Condition=" '$(PublishInstallerBaseVersion)' == 'true'">
<!--
This target is defined in eng/targets/Packaging.targets and Npm.Common.targets and included in every C#, F#,
and npm project. We use Microsoft.JSInterop.JS.npmproj because it is shipping (we need a stable
and npm project. We use Microsoft.JSInterop.JS.nodeproj because it is shipping (we need a stable
version string to use for productVersion.txt), and because it won't break when the SDK requires a newer
desktop MSBuild than exists on the build machine.
-->
<MSBuild Projects="$(RepoRoot)src\JSInterop\Microsoft.JSInterop.JS\src\Microsoft.JSInterop.JS.npmproj"
<MSBuild Projects="$(RepoRoot)src\JSInterop\Microsoft.JSInterop.JS\src\Microsoft.JSInterop.JS.nodeproj"
Properties="DisableYarnCheck=true;ExcludeFromBuild=false"
Targets="_GetPackageVersionInfo">
<Output TaskParameter="TargetOutputs" ItemName="_ResolvedProductVersionInfo" />
Expand Down
1 change: 1 addition & 0 deletions eng/RequiresDelayedBuildProjects.props
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,6 @@
<RequiresDelayedBuild Include="$(RepoRoot)src\Grpc\JsonTranscoding\test\testassets\IntegrationTestsWebsite\IntegrationTestsWebsite.csproj" />
<RequiresDelayedBuild Include="$(RepoRoot)src\Grpc\JsonTranscoding\test\testassets\Sandbox\Sandbox.csproj" />
<RequiresDelayedBuild Include="$(RepoRoot)src\ProjectTemplates\test\Templates.Blazor.Tests\Templates.Blazor.Tests.csproj" />
<RequiresDelayedBuild Include="$(RepoRoot)eng\Npm.Workspace.FunctionalTests.nodeproj" />
</ItemGroup>
</Project>
2 changes: 1 addition & 1 deletion eng/Tools.props
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
This is here to workaround flakiness in the NuGet SDK resolver in MSBuild. Arcade will run a pre-restore for
these packages. This works more consistently than the SDK resolution which uses global.json. Without this
here, we see regular failures with 'error MSB4236: The SDK 'Yarn.MSBuild' specified could not be found.'
Since this project is evaluated before .npmproj files are loaded, this should cause the package to end up in
Since this project is evaluated before .nodeproj files are loaded, this should cause the package to end up in
the NuGet cache ahead of time. This is not needed in source build.
-->
<PackageReference Include="Yarn.MSBuild" Version="1.22.10" />
Expand Down
Loading

0 comments on commit 8c27269

Please sign in to comment.