Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AML - Idle Shutdown test #2475

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
a8ee9ab
initial ML updates
batemansogq Oct 6, 2023
b6ea209
initial AML changes
batemansogq Oct 9, 2023
1f3ec94
corrected ML test
batemansogq Oct 9, 2023
209591f
corrected ML JSON, backing out test data
batemansogq Oct 9, 2023
86327ce
Merge pull request #1 from batemansogq/AML
batemansogq Oct 9, 2023
227f169
finalised md doco for ML
batemansogq Oct 9, 2023
c905e4d
inital ML disable local rule
batemansogq Oct 9, 2023
5061fae
ML disable admin rule
batemansogq Oct 10, 2023
191afa3
Merge pull request #2 from batemansogq/AML
batemansogq Oct 10, 2023
63a0d9c
ML compute vnet added, md update on hyperlinks
batemansogq Oct 10, 2023
02096ac
links updated for ML rules
batemansogq Oct 11, 2023
748d00a
update ML compute test ref for extend to workspaces
batemansogq Oct 11, 2023
c29ba4c
ML Workspace rule added, ML titles added to mds
batemansogq Oct 12, 2023
610968d
text upds
batemansogq Oct 12, 2023
d87d679
ML Wks pub access rule added. Refs updated
batemansogq Oct 12, 2023
bb864e4
ML Wkspace md updated
batemansogq Oct 12, 2023
c03f0c8
Merge pull request #3 from batemansogq/AML
batemansogq Oct 12, 2023
d80b591
Minor updates and tweaks
BernieWhite Oct 15, 2023
c45c515
Merge branch 'main' into main
BernieWhite Oct 15, 2023
9dbc7c6
Other minor updates
BernieWhite Oct 15, 2023
c960bb3
Merge branch 'Azure:main' into main
batemansogq Oct 17, 2023
adece5c
Merge branch 'Azure:main' into main
batemansogq Oct 17, 2023
1f924ad
Update docs/en/rules/Azure.ML.WrkspPublicAccess.md
batemansogq Oct 17, 2023
1717e72
Update docs/en/rules/Azure.ML.ComputeIdleShutdown.md
batemansogq Oct 17, 2023
b743b02
Update docs/en/rules/Azure.ML.WrkspPublicAccess.md
batemansogq Oct 17, 2023
cdcb7c0
Update docs/en/rules/Azure.ML.ComputeVnet.md
batemansogq Oct 17, 2023
f4905c5
Update src/PSRule.Rules.Azure/rules/Azure.ML.Rule.yaml
batemansogq Oct 17, 2023
5792480
Update src/PSRule.Rules.Azure/rules/Azure.ML.Rule.yaml
batemansogq Oct 17, 2023
8d3c52a
Merge branch 'main' into main
BernieWhite Oct 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions docs/en/rules/Azure.ML.ComputeIdleShutdown.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
reviewed: 2023-10-06
severity: Critical
pillar: Cost Optimization
category: Provision
resource: ML
online version: https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.ML.ComputeIdleShutdown/
---

# ML Compute Idle Shutdown

## SYNOPSIS

Use ML - Compute Instances configured for idle shutdown.
batemansogq marked this conversation as resolved.
Show resolved Hide resolved

## DESCRIPTION

Machine Learning uses compute instances as a training or inference compute for development and testing. It's similar to a virtual machine on the cloud.

To avoid getting charged for a compute instance that is switched on but not being actively used, you can configure when to automatically shut down compute instances due to inactivity.
batemansogq marked this conversation as resolved.
Show resolved Hide resolved

## RECOMMENDATION

Consider configuring ML - Compute Instances to automatically shutdown after a period of idle use as part of a broader cost optimization strategy.

## EXAMPLES

### Configure with Azure template

To deploy an ML - compute instance that complies with this rule:

- Provide a idle shutdown time as an ISO 8601 format string, i.e. 15mins = "PT15M"
- Define the "idleTimeBeforeShutdown" as this value

For example:

```json
{
"type": "Microsoft.MachineLearningServices/workspaces/computes",
"apiVersion": "2023-04-01",
"name": "[format('{0}/{1}', 'example-ws', parameters('name'))]",
"location": "[parameters('location')]",
"properties": {
"managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', 'example-mg')]",
"computeType": "ComputeInstance",
"properties": {
"vmSize": "[parameters('vmSize')]",
"idleTimeBeforeShutdown": "[parameters('idleTimeBeforeShutdown')]"
batemansogq marked this conversation as resolved.
Show resolved Hide resolved
}
}
}
```

### Configure with Bicep

To deploy an ML - compute instance that complies with this rule:

- Provide a idle shutdown time as an ISO 8601 format string, i.e. 15mins = 'PT15M'
- Define the "idleTimeBeforeShutdown" as this value

For example:

```bicep
resource aml_compute_instance 'Microsoft.MachineLearningServices/workspaces/computes@2023-04-01' ={
name: '${mlWorkspace.name}/${name}'
location: location
properties:{
managedResourceGroupId: managedRg.id
computeType: 'ComputeInstance'
properties: {
vmSize: vmSize
idleTimeBeforeShutdown: idleTimeBeforeShutdown // this must be a string in ISO 8601 format
batemansogq marked this conversation as resolved.
Show resolved Hide resolved
}
}
}

## LINKS

- [Configure idle shutdown](https://learn.microsoft.com/azure/machine-learning/how-to-create-compute-instance?view=azureml-api-2&tabs=azure-cli#configure-idle-shutdown)
- [ML - Compute objects](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-bicep#compute-objects)
- [ML - Workspaces](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/2023-04-01/workspaces?pivots=deployment-language-bicep)
- [ML Compute](https://learn.microsoft.com/azure/machine-learning/azure-machine-learning-glossary?view=azureml-api-2#compute)
- [AI + Machine Learning cost estimates](https://learn.microsoft.com/azure/well-architected/cost/provision-ai-ml)
91 changes: 91 additions & 0 deletions docs/en/rules/Azure.ML.ComputeVnet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
reviewed: 2023-10-10
severity: Critical
pillar: Security
category: Networking
resource: ML
online version: https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.ML.ComputeVnet/
---

# ML Compute hosted in VNet

## SYNOPSIS

Azure Machine Learning Computes should be hosted in a virtual network (VNet).

## DESCRIPTION

Azure Virtual Networks (VNets) provide enhanced security and isolation for your Azure Machine Learning Compute Clusters and Instances, as well as subnets, access control policies, and other features to further restrict access. When a compute is configured with a virtual network, it is not publicly addressable and can only be accessed from virtual machines and applications within the virtual network.

## RECOMMENDATION

ML - Compute should be hosted in a virtual network (VNet) as part of a broader security strategy.
batemansogq marked this conversation as resolved.
Show resolved Hide resolved

## EXAMPLES

### Configure with Azure template

To deploy an ML - compute that complies with this rule:

- update the compute properties to reference a specific subnet.

For example:

```json

{
"type": "Microsoft.MachineLearningServices/workspaces/computes",
"apiVersion": "2023-04-01",
"name": "[format('{0}/{1}', 'example-ws', parameters('name'))]",
"location": "[parameters('location')]",
"properties": {
"managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', 'example-rg')]",
"computeType": "[parameters('computeType')]",
"properties": {
"vmSize": "[parameters('vmSize')]",
"subnet": {
"id": "[parameters('subnetId')]"
}
}
}
}

```

### Configure with Bicep

To deploy an ML - compute that complies with this rule:

- update the compute properties to reference a specific subnet.

For example:

```bicep

resource aml_compute_instance 'Microsoft.MachineLearningServices/workspaces/computes@2023-04-01' ={
name: '${mlWorkspace.name}/${name}'
location: location

properties:{
managedResourceGroupId: managedRg.id
computeType: ComputeType
properties: {
vmSize: vmSize
subnet:{
id: subnet.id
}
}
}
}
```


## LINKS

- [Managed compute in a managed virtual network](https://learn.microsoft.com/azure/machine-learning/how-to-managed-network-compute?view=azureml-api-2&tabs=azure-cli)
- [ML - Network security and isolation](https://learn.microsoft.com/azure/machine-learning/concept-enterprise-security?view=azureml-api-2#network-security-and-isolation)
- [ML - Compute objects](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-bicep#resource-format)
- [ML - Workspaces](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/2023-04-01/workspaces?pivots=deployment-language-bicep)
- [ML Compute](https://learn.microsoft.com/azure/machine-learning/azure-machine-learning-glossary?view=azureml-api-2#compute)
- [WAF - Azure services for securing network connectivity](https://learn.microsoft.com/azure/well-architected/security/design-network-connectivity)
batemansogq marked this conversation as resolved.
Show resolved Hide resolved

85 changes: 85 additions & 0 deletions docs/en/rules/Azure.ML.DisableLocalAuth.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
reviewed: 2023-10-10
severity: Critical
pillar: Security
category: Identity and Access Management
resource: ML
online version: https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.ML.DisableLocalAuth/
---

# ML Compute has local authentication disabled

## SYNOPSIS

Azure Machine Learning compute resources should have local authentication methods disabled.

## DESCRIPTION

Disabling local authentication methods improves security by ensuring that Machine Learning Computes require Azure Active Directory identities exclusively for authentication.

## RECOMMENDATION

ML - Compute should be configured with local authentication disabled as part of a broader security strategy.

## EXAMPLES

### Configure with Azure template

To deploy an ML - compute that complies with this rule:

- Set the `disableLocalAuth` property value to true.

For example:

```json

{
"type": "Microsoft.MachineLearningServices/workspaces/computes",
"apiVersion": "2023-04-01",
"name": "[format('{0}/{1}', 'example-ws', parameters('name'))]",
"location": "[parameters('location')]",
"properties": {
"managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', 'example-rg')]",
"computeType": "[parameters('computeType')]",
"disableLocalAuth": true,
"properties": {
"vmSize": "[parameters('vmSize')]",
}
}
}

```

### Configure with Bicep

To deploy an ML - compute that complies with this rule:

- Set the `disableLocalAuth` property value to `true`.

For example:

```bicep

resource aml_compute_instance 'Microsoft.MachineLearningServices/workspaces/computes@2023-04-01' ={
name: '${mlWorkspace.name}/${name}'
location: location

properties:{
managedResourceGroupId: managedRg.id
computeType: ComputeType
disableLocalAuth: true
properties: {
vmSize: vmSize
}
}
}
```

## LINKS

- [Disable local authentication](https://learn.microsoft.com/azure/machine-learning/how-to-integrate-azure-policy?view=azureml-api-2#disable-local-authentication)
- [ML - Compute objects](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-bicep#resource-format)
- [ML - Workspaces](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/2023-04-01/workspaces?pivots=deployment-language-bicep)
- [ML Compute](https://learn.microsoft.com/azure/machine-learning/azure-machine-learning-glossary?view=azureml-api-2#compute)
- [Azure Policy Regulatory Compliance controls for Azure Machine Learning](https://learn.microsoft.com/azure/machine-learning/security-controls-policy?view=azureml-api-2)
- [WAF - Authentication with Azure AD](https://learn.microsoft.com/azure/well-architected/security/design-identity-authentication)
95 changes: 95 additions & 0 deletions docs/en/rules/Azure.ML.WrkspPublicAccess.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
reviewed: 2023-10-12
severity: Critical
pillar: Security
category: Networking
resource: ML
online version: https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.ML.WrkspPublicAccess/
---

# ML Workspace has public access disabled

## SYNOPSIS

Disable public network access from a ML - Workspace.

## DESCRIPTION

Disabling public network access improves security by ensuring that the Machine Learning Workspaces aren't exposed on the public internet. You can control exposure of your workspaces by creating private endpoints instead.

batemansogq marked this conversation as resolved.
Show resolved Hide resolved
## RECOMMENDATION

Consider setting the 'publicNetworkAccess' parameter of the Workspace properties to "Disabled", as part of a broader security strategy.

batemansogq marked this conversation as resolved.
Show resolved Hide resolved
## EXAMPLES

### Configure with Azure template

To deploy an ML - Workspace that complies with this rule:

- update the 'publicNetworkAccess' parameter of the Workspace properties to "Disabled".

For example:

```json
{
"type": "Microsoft.MachineLearningServices/workspaces",
"apiVersion": "2023-04-01",
"name": "[parameters('name')]",
"location": "[parameters('location')]",
"sku": {
"name": "basic",
"tier": "basic"
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"friendlyName": "[parameters('name')]",
"keyVault": "[resourceId('Microsoft.KeyVault/vaults', parameters('KeyVaultName'))]",
"storageAccount": "[resourceId('Microsoft.Storage/storageAccounts', parameters('StorageAccountName'))]",
"applicationInsights": "[resourceId('Microsoft.Insights/components', parameters('AppInsightsName'))]",
"containerRegistry": "[resourceId('Microsoft.ContainerRegistry/registries', parameters('ContainerRegistryName'))]",
"publicNetworkAccess": "Disabled"
}

```

### Configure with Bicep

To deploy an ML - Workspace that complies with this rule:

- update the 'publicNetworkAccess' parameter of the Workspace properties to 'Disabled'.

For example:

```bicep
resource Ml_Workspace 'Microsoft.MachineLearningServices/workspaces@2023-04-01' = {
name: name
location: location
sku: {
name: 'basic'
tier: 'basic'
}
identity: {
type: 'SystemAssigned'
}
properties: {
friendlyName: friendlyName
keyVault: KeyVault.id
storageAccount: StorageAccount.id
applicationInsights: AppInsights.id
containerRegistry: ContainerRegistry.id
publicNetworkAccess: 'Disabled'
}
}

```

## LINKS

- [Configure a private endpoint for an Azure Machine Learning workspace](https://learn.microsoft.com/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2&tabs=cli)
- [ML - Public access to Workspaces](https://learn.microsoft.com/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2&tabs=required%2Cpe%2Ccli#public-access-to-workspace)
- [ML - Workspaces](https://learn.microsoft.com/azure/templates/microsoft.machinelearningservices/workspaces?pivots=deployment-language-bicep#workspaceproperties)
- [Security and governance for ML](https://learn.microsoft.com/azure/machine-learning/concept-enterprise-security?view=azureml-api-2)
batemansogq marked this conversation as resolved.
Show resolved Hide resolved
- [WAF - Azure services for securing network connectivity](https://learn.microsoft.com/azure/well-architected/security/design-network-connectivity)
Loading
Loading