generated from adrienaury/go-template
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* docs: re-identification issue * test: re-identification function * test: package re-identification * test: re-identification dataset * test: doc TestReidentification * test: similarity metric on 2 dataset * refactor: package reidentification * feat: add reidentification in sigo * docs: example of trees reidentification * docs: 2nde example reidentification * docs: 2nde example reidentification * docs: 3rd example reidentification * docs: prevent re-identification * docs: prevent reidentification * docs: prevent reidentification * feat: sigo reidentification * feat: sigo reidentification * docs: add comments to functions * feat: sigo reidentification * refactor: define distance functions * feat: sigo reidentification * feat: sigo reidentification * feat: add function to scale data * feat: scale data before compute distance * docs: add README part reidentification * feat: add flag threshold reidentification * feat: add threshold to reidentification * docs: update README reidentification example * docs: update README reidentification example * docs: README explain reidentification * feat: add similarity score in output * docs: README add reidentification section * feat: add logs * feat: add logs * feat: add logs * docs: add reidentification tests * test: add benchmarck tests * feat: add reidentification in anonymizer * feat: add reidentification in anonymizer * feat: add arguments to anonymizer * feat: add checks for re-identification * test: add unit tests * docs: update README section reidentification * docs: remove old tests in example * docs: update README * docs: update README * docs: update README * docs: use LaTeX style syntax for formulas * docs: use LaTeX style syntax for formulas * docs: use LaTeX style syntax for formulas * docs: use LaTeX style syntax for formulas * docs: use LaTeX style syntax for formulas * test: add zerolog on benchmarck test --------- Co-authored-by: Youen Péron <[email protected]>
- Loading branch information
Showing
32 changed files
with
1,312 additions
and
104 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -258,6 +258,186 @@ DataSet after sequencing: | |
|
|
||
| Dates can be easily transformed into a sequence of floats, but one can imagine categories like colors, origin (if not a sensitive value), or even genders. | ||
|
|
||
| ## REIDENTIFICATION | ||
|
|
||
| With the evolution of information technologies that make it possible to link data from different sources, it is almost impossible to guarantee an anonymization that would offer a zero risk of re-identification. | ||
|
|
||
| **Re-identification Definition :** A process (or algorithm) that takes an anonymized dataset and related knowledge as input and seeks to match the anonymized data with real-world individuals. | ||
|
|
||
| Let's take as an example a very simple dataset that you can find in the `original.json` file in `examples/re-identification`. | ||
|
|
||
| ```json | ||
| {"id": 1, "x": 5, "y": 6, "z":"a"} | ||
| {"id": 2, "x": 3, "y": 7, "z":"a"} | ||
| {"id": 3, "x": 4, "y": 4, "z":"c"} | ||
| {"id": 4, "x": 2, "y": 10, "z":"b"} | ||
| {"id": 5, "x": 8, "y": 4, "z":"a"} | ||
| {"id": 6, "x": 8, "y": 10, "z":"a"} | ||
| ... | ||
| ``` | ||
|
|
||
| And suppose that we have 2 quasi-identifiers: `x` and `y` and as sensitive data the variable `z`. Anonymize the dataset using `sigo`, we use sigo's default settings **k=3** and **l=1** with the **meanAggregation** method : | ||
|
|
||
| ```console | ||
| sigo -q x,y -s z -a meanAggregation -i cluster < original.json > anonymized.json | ||
| ``` | ||
|
|
||
| ```json | ||
| {"id":1,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":2,"x":3,"y":7,"z":"a","cluster":1} | ||
| {"id":3,"x":3,"y":7,"z":"c","cluster":1} | ||
| {"id":4,"x":3,"y":7,"z":"b","cluster":1} | ||
| {"id":5,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":6,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| ... | ||
| ``` | ||
|
|
||
| **Objective :** Identify for each individual in the original dataset (data from the open data) whether an anonymized individual is similar to him assuming the worst case scenario, i.e. the attacker has the original dataset but not the sensitive data. | ||
|
|
||
| The data that the attacker has is in the `openData.json` file in `examples/re-identification`. | ||
|
|
||
| ```json | ||
| {"id": 1, "x": 5, "y": 6} | ||
| {"id": 2, "x": 3, "y": 7} | ||
| {"id": 3, "x": 4, "y": 4} | ||
| {"id": 4, "x": 2, "y": 10} | ||
| {"id": 5, "x": 8, "y": 4} | ||
| {"id": 6, "x": 8, "y": 10} | ||
| ... | ||
| ``` | ||
|
|
||
|  | ||
|
|
||
| Our method of re-identification is to find the closest or most similar individuals. | ||
|
|
||
| This approach depends greatly on the concepts of distance and similarity (more details in èxample/re-identification/concept). | ||
|
|
||
| ### Approach | ||
|
|
||
| - 1st step: | ||
|
|
||
| Merge the anonymized data and the data from the open data, then add a binary attribute *original* which indicates the origin of the individual (`0` if it is the anonymized data and `1` if it is the external data to be re-identified). | ||
|
|
||
|  | ||
|
|
||
| The files are located in the folder `examples/re-identification`. | ||
|
|
||
| ```console | ||
| cat anonymized.json openData.json > merged.json | ||
| ``` | ||
|
|
||
| ``` json | ||
| ... | ||
| {"original":0,"id":22,"x":16.67,"y":18.33,"z":"c"} | ||
| {"original":0,"id":23,"x":16.67,"y":18.33,"z":"b"} | ||
| {"original":0,"id":24,"x":19.67,"y":17.67,"z":"b"} | ||
| {"original":1,"id": 1, "x": 5, "y": 6} | ||
| {"original":1,"id": 2, "x": 3, "y": 7} | ||
| {"original":1,"id": 3, "x": 4, "y": 4} | ||
| ... | ||
| ``` | ||
|
|
||
| - 2nd step: | ||
|
|
||
| Use sigo to form clusters of similar individuals. | ||
|
|
||
| ```console | ||
| sigo -q x,y -s original -k 6 -l 2 -i cluster < merged.json | ||
| ``` | ||
|
|
||
| ``` json | ||
| {"id":2,"original":0,"x":3,"y":7,"z":"a","cluster":1} | ||
| {"id":2,"original":1,"x":3,"y":7,"z":null,"cluster":1} | ||
| {"id":3,"original":0,"x":3,"y":7,"z":"c","cluster":1} | ||
| {"id":3,"original":1,"x":4,"y":4,"z":null,"cluster":1} | ||
| {"id":4,"original":1,"x":2,"y":10,"z":null,"cluster":1} | ||
| {"id":4,"original":0,"x":3,"y":7,"z":"b","cluster":1} | ||
| {"id":1,"original":1,"x":5,"y":6,"z":null,"cluster":2} | ||
| {"id":1,"original":0,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":5,"original":0,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":5,"original":1,"x":8,"y":4,"z":null,"cluster":2} | ||
| {"id":6,"original":0,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":6,"original":1,"x":8,"y":10,"z":null,"cluster":2} | ||
| ... | ||
| ``` | ||
|
|
||
| - 3rd step: | ||
|
|
||
| - if in a cluster the sensitive data has the same value then we can re-identify the original individuals with this sensitive data. | ||
| - if in a cluster the sensitive data is not the same but the anonymized individuals are the same then we are not able to re-identify the original individuals. | ||
| - if in a cluster the sensitive data is not the same and the anonymized individuals are not the same then a distance computation will be performed to try to re-identify the individuals. | ||
|
|
||
|  | ||
|
|
||
| Below is the use of `sigo` for re-identification. | ||
|
|
||
| ```console | ||
| sigo -q x,y -s original -k 6 -l 2 -a reidentification --args z < merged.json | ||
| ``` | ||
|
|
||
| ```json | ||
| {"id":1,"original":1,"x":5,"y":6,"z":"a"} | ||
| {"id":1,"original":0,"x":7,"y":6.67,"z":"a"} | ||
| {"id":2,"original":0,"x":3,"y":7,"z":"a"} | ||
| {"id":2,"original":1,"x":3,"y":7,"z":null} | ||
| {"id":3,"original":0,"x":3,"y":7,"z":"c"} | ||
| {"id":3,"original":1,"x":4,"y":4,"z":null} | ||
| {"id":4,"original":1,"x":2,"y":10,"z":null} | ||
| {"id":4,"original":0,"x":3,"y":7,"z":"b"} | ||
| {"id":5,"original":0,"x":7,"y":6.67,"z":"a"} | ||
| {"id":5,"original":1,"x":8,"y":4,"z":"a"} | ||
| {"id":6,"original":0,"x":7,"y":6.67,"z":"a"} | ||
| {"id":6,"original":1,"x":8,"y":10,"z":"a"} | ||
| {"id":7,"original":1,"x":3,"y":16,"z":"a"} | ||
| {"id":7,"original":0,"x":4.33,"y":17.67,"z":"a"} | ||
| {"id":8,"original":1,"x":7,"y":19,"z":null} | ||
| {"id":8,"original":0,"x":8,"y":15.67,"z":"a"} | ||
| {"id":9,"original":0,"x":4.33,"y":17.67,"z":"a"} | ||
| {"id":9,"original":1,"x":6,"y":18,"z":"a"} | ||
| {"id":10,"original":0,"x":4,"y":18,"z":"b"} | ||
| {"id":10,"original":1,"x":4,"y":19,"z":"b"} | ||
| {"id":11,"original":1,"x":7,"y":14,"z":null} | ||
| {"id":11,"original":0,"x":8,"y":15.67,"z":"c"} | ||
| {"id":12,"original":0,"x":8,"y":15.67,"z":"c"} | ||
| {"id":12,"original":1,"x":10,"y":14,"z":null} | ||
| {"id":13,"original":1,"x":15,"y":5,"z":null} | ||
| {"id":13,"original":0,"x":16,"y":6,"z":"c"} | ||
| {"id":14,"original":1,"x":15,"y":7,"z":null} | ||
| {"id":14,"original":0,"x":16,"y":6,"z":"b"} | ||
| {"id":15,"original":1,"x":11,"y":9,"z":null} | ||
| {"id":15,"original":0,"x":12.33,"y":6,"z":"b"} | ||
| {"id":16,"original":1,"x":12,"y":3,"z":null} | ||
| {"id":16,"original":0,"x":12.33,"y":6,"z":"a"} | ||
| {"id":17,"original":0,"x":16,"y":6,"z":"c"} | ||
| {"id":17,"original":1,"x":18,"y":6,"z":null} | ||
| {"id":18,"original":0,"x":12.33,"y":6,"z":"c"} | ||
| {"id":18,"original":1,"x":14,"y":6,"z":null} | ||
| {"id":19,"original":0,"x":19.67,"y":17.67,"z":"b"} | ||
| {"id":19,"original":1,"x":20,"y":20,"z":"b"} | ||
| {"id":20,"original":0,"x":16.67,"y":18.33,"z":"c"} | ||
| {"id":20,"original":1,"x":18,"y":19,"z":null} | ||
| {"id":21,"original":0,"x":19.67,"y":17.67,"z":"b"} | ||
| {"id":21,"original":1,"x":20,"y":18,"z":"b"} | ||
| {"id":22,"original":0,"x":16.67,"y":18.33,"z":"c"} | ||
| {"id":22,"original":1,"x":18,"y":18,"z":null} | ||
| {"id":23,"original":1,"x":14,"y":18,"z":null} | ||
| {"id":23,"original":0,"x":16.67,"y":18.33,"z":"b"} | ||
| {"id":24,"original":1,"x":19,"y":15,"z":"b"} | ||
| {"id":24,"original":0,"x":19.67,"y":17.67,"z":"b"} | ||
| ``` | ||
|
|
||
| ### Usage for sigo reidentification | ||
|
|
||
| These flags must be used for re-identification : | ||
|
|
||
| - `--quasi-identifier,-q <strings>`, this flag lists the attributes (quasi-identifiers) of datasets. | ||
| - `--sensitive,-s original`, this flag tells sigo to make the difference between anonymized data and original data to be re-identified. | ||
| - `--l-value,-l 2`, the l-diversity parameter is set to 2 to have at least one anonymized data and one original data in the same cluster. | ||
| - `--anonymizer,-a reidentification`, this flag allows to run the re-identification. | ||
| - `--args <strings>`, this flag lists the arguments to pass to the re-identification method, i.e. the list of sensitive attributes of the dataset. | ||
|
|
||
| The other `sigo` flags can be used in addition. | ||
|
|
||
| ## Contributors | ||
|
|
||
| - CGI France ✉[Contact support](mailto:[email protected]) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| {"id":1,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":2,"x":3,"y":7,"z":"a","cluster":1} | ||
| {"id":3,"x":3,"y":7,"z":"c","cluster":1} | ||
| {"id":4,"x":3,"y":7,"z":"b","cluster":1} | ||
| {"id":5,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":6,"x":7,"y":6.67,"z":"a","cluster":2} | ||
| {"id":7,"x":4.33,"y":17.67,"z":"a","cluster":3} | ||
| {"id":8,"x":8,"y":15.67,"z":"a","cluster":4} | ||
| {"id":9,"x":4.33,"y":17.67,"z":"a","cluster":3} | ||
| {"id":10,"x":4.33,"y":17.67,"z":"b","cluster":3} | ||
| {"id":11,"x":8,"y":15.67,"z":"c","cluster":4} | ||
| {"id":12,"x":8,"y":15.67,"z":"c","cluster":4} | ||
| {"id":13,"x":16,"y":6,"z":"c","cluster":6} | ||
| {"id":14,"x":16,"y":6,"z":"b","cluster":6} | ||
| {"id":15,"x":12.33,"y":6,"z":"b","cluster":5} | ||
| {"id":16,"x":12.33,"y":6,"z":"a","cluster":5} | ||
| {"id":17,"x":16,"y":6,"z":"c","cluster":6} | ||
| {"id":18,"x":12.33,"y":6,"z":"c","cluster":5} | ||
| {"id":19,"x":19.67,"y":17.67,"z":"b","cluster":8} | ||
| {"id":20,"x":16.67,"y":18.33,"z":"c","cluster":7} | ||
| {"id":21,"x":19.67,"y":17.67,"z":"b","cluster":8} | ||
| {"id":22,"x":16.67,"y":18.33,"z":"c","cluster":7} | ||
| {"id":23,"x":16.67,"y":18.33,"z":"b","cluster":7} | ||
| {"id":24,"x":19.67,"y":17.67,"z":"b","cluster":8} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| {"id":1,"x":5.938120851374984,"y":4.533965986177158,"z":"a"} | ||
| {"id":2,"x":2.680835241693462,"y":8.19425223836935,"z":"a"} | ||
| {"id":3,"x":3.4851832584055824,"y":5.668937832579665,"z":"c"} | ||
| {"id":4,"x":2.191579318404636,"y":5.768462278793522,"z":"b"} | ||
| {"id":5,"x":7.849763705115755,"y":6.176648218859225,"z":"a"} | ||
| {"id":6,"x":7.236515404399387,"y":4.889286506988984,"z":"a"} | ||
| {"id":7,"x":3.5463258052174105,"y":16.62968221276086,"z":"a"} | ||
| {"id":8,"x":7.166427916409467,"y":14.853870793985047,"z":"a"} | ||
| {"id":9,"x":5.313935236579784,"y":18.671614394540253,"z":"a"} | ||
| {"id":10,"x":3.525701395319161,"y":18.180632263551825,"z":"b"} | ||
| {"id":11,"x":7.596790801515696,"y":14.387014064867332,"z":"c"} | ||
| {"id":12,"x":8.371563137890895,"y":16.27598200732568,"z":"c"} | ||
| {"id":13,"x":15.9401941749371,"y":5.703704179313746,"z":"c"} | ||
| {"id":14,"x":15.912168810000395,"y":6.760290278787764,"z":"b"} | ||
| {"id":15,"x":12.056370844747219,"y":7.593606438643265,"z":"b"} | ||
| {"id":16,"x":12.399920711511685,"y":3.289488817191873,"z":"a"} | ||
| {"id":17,"x":17.281844840528116,"y":5.019956467119892,"z":"c"} | ||
| {"id":18,"x":11.670023549374136,"y":8.978933759942723,"z":"c"} | ||
| {"id":19,"x":19.92395414411107,"y":16.401105715881105,"z":"b"} | ||
| {"id":20,"x":14.35294166396883,"y":18.172668663775376,"z":"c"} | ||
| {"id":21,"x":19.987901071971024,"y":19.864786326688186,"z":"b"} | ||
| {"id":22,"x":17.20752528905433,"y":18.546736879113336,"z":"c"} | ||
| {"id":23,"x":14.442379076836225,"y":18.044346789802102,"z":"b"} | ||
| {"id":24,"x":19.476769170963276,"y":18.014504697069313,"z":"b"} |
Oops, something went wrong.