Skip to content
This repository has been archived by the owner on Mar 31, 2023. It is now read-only.

Fix race condition in subnet deletion API #663

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
9645772
Documentation update
pkommoju May 12, 2021
183c36a
Apply changes suggested in review
pkommoju May 22, 2021
4896515
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju May 26, 2021
f170e1e
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju May 28, 2021
bcbf7b1
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju May 29, 2021
efc5b6e
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 1, 2021
c785697
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 8, 2021
d213c03
Merge branch 'master' of https://github.com/pkommoju/alcor
Jun 8, 2021
0f56454
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 8, 2021
4a2a382
Merge branch 'master' of https://github.com/pkommoju/alcor
pkommoju Jun 9, 2021
7f93fe3
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 10, 2021
8d6dd0e
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 15, 2021
b19d428
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
kevin-zhonghao Jun 16, 2021
539d0c0
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jun 22, 2021
6a9a1c6
Merge branch 'master' of https://github.com/pkommoju/alcor
pkommoju Jun 22, 2021
19430b2
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 1, 2021
1d7ef7e
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 2, 2021
cd1fec6
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 6, 2021
e079a44
Fix race condition in VPC Mgr delete subnet API
Jul 9, 2021
4831ad6
Set subnset ip version as IPv4
Jul 9, 2021
3c378e0
Merge branch 'master' into fix/subnet_concurrent_delete_issue
Jul 9, 2021
4cb934e
Fix varible type
Jul 9, 2021
ad62ef0
Add some logs
Jul 9, 2021
f756eef
Add more logs
Jul 9, 2021
8160d4a
Revert the wrong fix to the non-gateway port count check
Jul 9, 2021
639a0cd
Fix commit transaction from different thread issue
Jul 9, 2021
85de8fb
Update ignite config xml to support transaction in VpcRepository
Jul 9, 2021
d6ed075
Fix Ip mgr ignite misconfiguration
Jul 13, 2021
e57abe4
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 20, 2021
1b63054
Add missing file
pkommoju Jul 21, 2021
fe81008
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 21, 2021
016aaf5
Add create ignite client with cache config and support in ip mgr
Jul 22, 2021
118dd53
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Jul 23, 2021
7d9a2e5
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Aug 2, 2021
3884a44
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Aug 4, 2021
0b5cd11
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Aug 6, 2021
2ae9045
Merge from master
Aug 20, 2021
257ef71
Make all currently used caches transactional
pkommoju Aug 26, 2021
c303e0b
Merge branch 'master' of https://github.com/futurewei-cloud/alcor
pkommoju Aug 26, 2021
46b79f8
Merge branch 'master' into all_txn
pkommoju Aug 26, 2021
c7809b3
Make more caches to Transactional atomicity to fix subnet deletion pr…
pkommoju Aug 26, 2021
70e9251
Address code review comments
pkommoju Aug 27, 2021
06bc414
Merge branch 'all_txn' into fix/subnet_concurrent_delete_issue
pkommoju Aug 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -467,18 +467,10 @@ public ResponseId deleteSubnetState(@PathVariable String projectId, @PathVariabl
}

String rangeId = null;
String ipV4RangeId = subnetEntity.getIpV4RangeId();
String ipV6RangeId = subnetEntity.getIpV6RangeId();
if (ipV4RangeId != null) {
rangeId = ipV4RangeId;
if (subnetEntity.getIpVersion() == 6) {
rangeId = subnetEntity.getIpV6RangeId();
} else {
rangeId = ipV6RangeId;
}

// TODO: check if there is any gateway / non-gateway port for the subnet, waiting for PM new API
Boolean checkIfAnyNoneGatewayPortInSubnet = this.subnetService.checkIfAnyPortInSubnet(projectId, subnetId);
if (checkIfAnyNoneGatewayPortInSubnet) {
throw new HavePortInSubnet();
rangeId = subnetEntity.getIpV4RangeId();
}

// check if subnet bind any router
Expand All @@ -494,7 +486,13 @@ public ResponseId deleteSubnetState(@PathVariable String projectId, @PathVariabl
logger.warn(e.getMessage());
}

// TODO: delete gateway port in port manager. Temporary solution, need PM fix issue
// check if there is any non-gateway port for the subnet
boolean checkIfAnyNoneGatewayPortInSubnet = this.subnetService.checkIfAnyNonGatewayPortInSubnet(projectId, subnetEntity);
if (checkIfAnyNoneGatewayPortInSubnet) {
throw new HaveNonGatewayPortInSubnet();
}

// delete gateway port in port manager
GatewayPortDetail gatewayPortDetail = subnetEntity.getGatewayPortDetail();
if (gatewayPortDetail != null) {
this.subnetToPortManagerService.deleteGatewayPort(projectId, gatewayPortDetail.getGatewayPortId());
Expand All @@ -508,7 +506,7 @@ public ResponseId deleteSubnetState(@PathVariable String projectId, @PathVariabl

this.subnetDatabaseService.deleteSubnet(subnetId);

} catch (ParameterNullOrEmptyException | HavePortInSubnet | SubnetBindRouter e) {
} catch (ParameterNullOrEmptyException | HaveNonGatewayPortInSubnet | SubnetBindRouter e) {
logger.error(e.getMessage());
throw new Exception(e);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ free of charge, to any person obtaining a copy of this software and associated d
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(code= HttpStatus.CONFLICT, reason="There is some ports in the subnet, we can Not delete subnet")
public class HavePortInSubnet extends Exception {
@ResponseStatus(code= HttpStatus.CONFLICT, reason="There is some customer ports in the subnet, we can Not delete subnet")
public class HaveNonGatewayPortInSubnet extends Exception {
}
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ public void fallbackOperation (AtomicReference<RouteWebJson> routeResponseAtomic
public void deleteSubnetIdInVpc (String subnetId, String projectId, String vpcId) throws Exception;

// check if there is any port in this subnet
public boolean checkIfAnyPortInSubnet (String projectId, String subnetId) throws SubnetIdIsNull;
public boolean checkIfAnyNonGatewayPortInSubnet(String projectId, SubnetEntity subnetEntity) throws SubnetIdIsNull;

// check if subnet bind any routes
public boolean checkIfSubnetBindAnyRouter(SubnetEntity subnetEntity);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ free of charge, to any person obtaining a copy of this software and associated d
import com.futurewei.alcor.web.entity.route.*;
import com.futurewei.alcor.web.entity.subnet.*;
import com.futurewei.alcor.web.entity.vpc.VpcWebJson;
import io.netty.util.internal.StringUtil;
import org.apache.commons.net.util.SubnetUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand Down Expand Up @@ -304,12 +305,15 @@ public void deleteSubnetIdInVpc(String subnetId, String projectId, String vpcId)
}

@Override
public boolean checkIfAnyPortInSubnet(String projectId, String subnetId) throws SubnetIdIsNull {
if (subnetId == null) {
public boolean checkIfAnyNonGatewayPortInSubnet(String projectId, SubnetEntity subnetEntity) throws SubnetIdIsNull {
if (subnetEntity == null || StringUtil.isNullOrEmpty(subnetEntity.getId())) {
throw new SubnetIdIsNull();
}
String portManagerServiceUrl = portUrl + "project/" + projectId + "/subnet-port-count/" + subnetId;
int portCount = restTemplate.getForObject(portManagerServiceUrl, Integer.class);

String portManagerServiceUrl = portUrl + "project/" + projectId + "/subnet-port-count/" + subnetEntity.getId();
int portCount = restTemplate.getForObject(portManagerServiceUrl, Integer.class);

logger.info("[checkIfAnyNonGatewayPortInSubnet]: portCount == " + portCount + " && subnetEntity.getGatewayPortId() = " + subnetEntity.getGatewayPortId());
if (portCount == 0) {
return false;
}
Expand All @@ -321,7 +325,7 @@ public boolean checkIfAnyPortInSubnet(String projectId, String subnetId) throws
public boolean checkIfSubnetBindAnyRouter(SubnetEntity subnetEntity) {

String attachedRouterId = subnetEntity.getAttachedRouterId();
if (attachedRouterId == null || attachedRouterId.equals("")){
if (attachedRouterId == null || attachedRouterId.equals("")) {
return false;
}

Expand All @@ -330,7 +334,7 @@ public boolean checkIfSubnetBindAnyRouter(SubnetEntity subnetEntity) {

@Override
@DurationStatistics
public boolean checkIfCidrOverlap(String cidr,String projectId, String vpcId) throws FallbackException, ResourceNotFoundException, ResourcePersistenceException, CidrNotWithinNetworkCidr, CidrOverlapWithOtherSubnets {
public boolean checkIfCidrOverlap(String cidr, String projectId, String vpcId) throws FallbackException, ResourceNotFoundException, ResourcePersistenceException, CidrNotWithinNetworkCidr, CidrOverlapWithOtherSubnets {

// get vpc and check with vpc cidr
VpcWebJson vpcWebJson = verifyVpcId(projectId, vpcId);
Expand All @@ -341,8 +345,7 @@ public boolean checkIfCidrOverlap(String cidr,String projectId, String vpcId) th
throw new CidrNotWithinNetworkCidr();
}
}



// get subnet list and check with subnets cidr
List<String> subnetIds = vpcWebJson.getNetwork().getSubnets();
for (String subnetId : subnetIds) {
Expand Down Expand Up @@ -552,7 +555,7 @@ public void deleteIPRangeInPIM(String rangeId) {
return;
}

String ipManagerCreateRangeUrl = ipUrl + "range/"+ rangeId;
String ipManagerCreateRangeUrl = ipUrl + "range/" + rangeId;
restTemplate.delete(ipManagerCreateRangeUrl);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ public class VpcController {
* @return vpc state
* @throws Exception
*/
@Rbac(resource ="vpc")
@Rbac(resource = "vpc")
@FieldFilter(type = VpcEntity.class)
@RequestMapping(
method = GET,
Expand Down Expand Up @@ -114,7 +114,7 @@ public VpcsWebJson createVpcStateBulk(@PathVariable String projectid, @RequestBo
* @return vpc state
* @throws Exception
*/
@Rbac(resource ="vpc")
@Rbac(resource = "vpc")
@RequestMapping(
method = POST,
value = {"/project/{projectid}/vpcs"})
Expand Down Expand Up @@ -194,7 +194,7 @@ public VpcWebJson createVpcState(@PathVariable String projectid, @RequestBody Vp
* @return vpc state
* @throws Exception
*/
@Rbac(resource ="vpc")
@Rbac(resource = "vpc")
@RequestMapping(
method = PUT,
value = {"/project/{projectid}/vpcs/{vpcid}"})
Expand Down Expand Up @@ -252,7 +252,7 @@ public VpcWebJson updateVpcStateByVpcId(@PathVariable String projectid, @PathVar
* @return network id
* @throws Exception
*/
@Rbac(resource ="vpc")
@Rbac(resource = "vpc")
@RequestMapping(
method = DELETE,
value = {"/project/{projectid}/vpcs/{vpcid}"})
Expand Down Expand Up @@ -290,16 +290,16 @@ public ResponseId deleteVpcStateByVpcId(@PathVariable String projectid, @PathVar
* @return Map<String, VpcWebResponseObject>
* @throws Exception
*/
@Rbac(resource ="vpc")
@Rbac(resource = "vpc")
@FieldFilter(type = VpcEntity.class)
@RequestMapping(
method = GET,
value = "/project/{projectId}/vpcs")
@DurationStatistics
public VpcsWebJson getVpcStatesByProjectId(@PathVariable String projectId) throws Exception {
Map<String, VpcEntity> vpcStates = null;
Map<String, String[]> requestParams = (Map<String, String[]>)request.getAttribute(QUERY_ATTR_HEADER);
requestParams = requestParams == null ? request.getParameterMap():requestParams;
Map<String, String[]> requestParams = (Map<String, String[]>) request.getAttribute(QUERY_ATTR_HEADER);
requestParams = requestParams == null ? request.getParameterMap() : requestParams;
Map<String, Object[]> queryParams =
ControllerUtil.transformUrlPathParams(requestParams, VpcEntity.class);

Expand Down Expand Up @@ -339,6 +339,7 @@ public Map getVpcCountAndAllVpcStates() throws CacheException {

/**
* Updates a network with subnet id
*
* @param projectid
* @param vpcid
* @param subnetid
Expand Down Expand Up @@ -381,11 +382,11 @@ public VpcWebJson addSubnetIdToVpcState(@PathVariable String projectid, @PathVar
}

return new VpcWebJson(inVpcState);

}

/**
* delete subnet id in a network
*
* @param projectid
* @param vpcid
* @param subnetid
Expand All @@ -398,35 +399,19 @@ public VpcWebJson addSubnetIdToVpcState(@PathVariable String projectid, @PathVar
@DurationStatistics
public VpcWebJson deleteSubnetIdInVpcState(@PathVariable String projectid, @PathVariable String vpcid, @PathVariable String subnetid) throws Exception {

VpcEntity inVpcState = new VpcEntity();
VpcEntity inVpcState = null;

try {
RestPreconditionsUtil.verifyParameterNotNullorEmpty(projectid);
RestPreconditionsUtil.verifyParameterNotNullorEmpty(vpcid);
RestPreconditionsUtil.verifyParameterNotNullorEmpty(subnetid);

inVpcState = this.vpcDatabaseService.getByVpcId(vpcid);
if (inVpcState == null) {
throw new ResourceNotFoundException("Vpc not found : " + vpcid);
}

List<String> subnets = inVpcState.getSubnets();
if (subnets == null || !subnets.contains(subnetid)) {
return new VpcWebJson(inVpcState);
}
subnets.remove(subnetid);

inVpcState.setSubnets(subnets);

this.vpcDatabaseService.addVpc(inVpcState);

inVpcState = this.vpcDatabaseService.getByVpcId(vpcid);
inVpcState = this.vpcDatabaseService.deleteSubnetIdInVpc(vpcid, subnetid);

} catch (ParameterNullOrEmptyException e) {
throw new Exception(e);
}

return new VpcWebJson(inVpcState);

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ free of charge, to any person obtaining a copy of this software and associated d
import com.futurewei.alcor.common.db.CacheException;
import com.futurewei.alcor.common.db.CacheFactory;
import com.futurewei.alcor.common.db.ICache;
import com.futurewei.alcor.common.db.Transaction;
import com.futurewei.alcor.common.db.repo.ICacheRepository;
import com.futurewei.alcor.common.logging.Logger;
import com.futurewei.alcor.common.logging.LoggerFactory;
Expand All @@ -43,6 +44,14 @@ public ICache<String, VpcEntity> getCache() {

private ICache<String, VpcEntity> cache;

public Transaction startTransaction() throws CacheException {
return cache.getTransaction().start();
}

public void commitTransaction() throws CacheException {
cache.getTransaction().commit();
}

@Autowired
public VpcRepository(CacheFactory cacheFactory) {
cache = cacheFactory.getCache(VpcEntity.class);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,18 @@ free of charge, to any person obtaining a copy of this software and associated d
import com.futurewei.alcor.common.db.CacheException;
import com.futurewei.alcor.common.db.ICache;
import com.futurewei.alcor.common.exception.DatabasePersistenceException;
import com.futurewei.alcor.common.exception.ResourceNotFoundException;
import com.futurewei.alcor.common.stats.DurationStatistics;
import com.futurewei.alcor.vpcmanager.dao.VpcRepository;
import com.futurewei.alcor.vpcmanager.service.VpcDatabaseService;
import com.futurewei.alcor.web.entity.vpc.VpcEntity;
import com.futurewei.alcor.web.entity.vpc.VpcWebJson;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.Map;

@Service
Expand All @@ -42,7 +45,7 @@ public class VpcDatabaseServiceImpl implements VpcDatabaseService {
public VpcEntity getByVpcId(String vpcId) {
try {
return this.vpcRepository.findItem(vpcId);
}catch (Exception e) {
} catch (Exception e) {
return null;
}
}
Expand Down Expand Up @@ -75,6 +78,35 @@ public void deleteVpc(String id) throws CacheException {
this.vpcRepository.deleteItem(id);
}

@Override
public VpcEntity deleteSubnetIdInVpc(String vpcId, String subnetId) throws ResourceNotFoundException, DatabasePersistenceException, CacheException {

VpcEntity currentVpcState = null;

try {
this.vpcRepository.startTransaction();
currentVpcState = getByVpcId(vpcId);
if (currentVpcState == null) {
throw new ResourceNotFoundException("Vpc not found : " + vpcId);
}

List<String> subnets = currentVpcState.getSubnets();
if (subnets == null || !subnets.contains(subnetId)) {
return currentVpcState;
}

subnets.remove(subnetId);
currentVpcState.setSubnets(subnets);
addVpc(currentVpcState);
} catch (ResourceNotFoundException | DatabasePersistenceException | CacheException e) {
throw e;
} finally {
this.vpcRepository.commitTransaction();
}

return currentVpcState;
}

@Override
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good but there is a convention about the naming. The convention is to call someCache.getTransaction().start(), .commit(), .rollback() instead of wrapping the transaction calls in a method and calling it. It would be better if this code follows the convention.

Another change is required for this to actually work. Ignite configuration declare that this cache must have atomicity mode of TRANSACTIONAL. Here is the declatration.

        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <!-- Set the cache name. -->
             <property name="name" value="com.futurewei.alcor.web.entity.vpc.VpcEntity"/>
            <!-- Set the cache mode. -->
            <property name="atomicityMode" value="TRANSACTIONAL"/>
            <!-- Other cache parameters. -->
            <property name="cacheMode" value="PARTITIONED"/>
         </bean>

In kubernetes/services/ignite_config.xml there is this section

under which there is tag . The above declaration should be added under the tag.

Every one needs this update to the ignite config file to not run into this issue, in addition the iginte database should be deleted. (default is <your_ignite_home>/work). If anyone has valuable data in the ignite DB, let me know we can delete just this cache.

If there are any questions about this extra work, ping me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pkommoju Thanks. We have updated the transaction naming and added a new cacheConfiguration for VpcRepo.

@DurationStatistics
public ICache<String, VpcEntity> getCache() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ public interface VpcDatabaseService {
public Map getAllVpcs (Map<String, Object[]> queryParams) throws CacheException;
public void addVpc (VpcEntity vpcState) throws DatabasePersistenceException;
public void deleteVpc (String id) throws CacheException;
public VpcEntity deleteSubnetIdInVpc(String vpcId, String subnetId) throws ResourceNotFoundException, DatabasePersistenceException, CacheException;
public ICache<String, VpcEntity> getCache ();

}
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ public SubnetEntity(String projectId, String vpcId, String id, String name, Stri
super(projectId, id, name, null);
this.vpcId = vpcId;
this.cidr = cidr;
this.ipVersion = 4;
}

public SubnetEntity(String projectId, String id, String name, String description, String vpcId,
Expand Down